Jul 15 19:28:38.008162 ip-10-0-142-40 systemd[1]: Starting Kubernetes Kubelet... Jul 15 19:28:38.451487 ip-10-0-142-40 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:28:38.451487 ip-10-0-142-40 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 19:28:38.451487 ip-10-0-142-40 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:28:38.451487 ip-10-0-142-40 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 19:28:38.451487 ip-10-0-142-40 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:28:38.454038 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.453954 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 19:28:38.457936 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457921 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:28:38.457936 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457937 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457941 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457944 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457946 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457951 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457955 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457958 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457961 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457964 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457967 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457969 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457977 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457981 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457984 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457986 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457989 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457992 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457994 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457997 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:28:38.458000 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.457999 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458002 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458005 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458008 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458011 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458013 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458016 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458018 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458021 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458024 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458026 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458029 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458031 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458034 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458036 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458038 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458041 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458043 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458045 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458048 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:28:38.458463 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458050 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458053 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458055 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458057 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458060 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458062 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458064 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458067 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458070 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458073 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458076 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458078 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458081 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458083 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458088 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458092 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458095 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458098 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458101 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:28:38.458956 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458103 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458106 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458108 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458111 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458114 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458116 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458118 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458121 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458124 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458126 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458128 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458131 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458136 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458139 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458142 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458145 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458147 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458149 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458152 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458158 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:28:38.459449 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458160 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458163 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458165 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458167 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458170 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458172 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458175 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458178 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458180 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458183 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458185 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458188 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458190 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458193 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458195 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458197 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458200 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458203 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458205 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458208 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:28:38.459941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458986 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.458998 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459001 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459004 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459007 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459009 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459012 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459015 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459018 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459022 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459025 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459028 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459031 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459033 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459036 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459038 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459041 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459043 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459046 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459048 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:28:38.460431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459051 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459053 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459056 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459058 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459061 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459063 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459065 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459068 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459070 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459073 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459075 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459077 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459080 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459085 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459088 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459091 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459093 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459095 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459098 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:28:38.460941 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459101 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459104 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459108 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459112 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459115 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459117 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459120 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459122 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459124 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459127 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459129 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459132 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459134 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459137 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459139 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459141 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459144 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459146 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459148 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:28:38.461391 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459151 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459153 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459156 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459158 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459161 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459163 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459166 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459169 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459172 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459175 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459178 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459180 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459182 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459185 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459188 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459190 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459193 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459195 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459197 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:28:38.461983 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459201 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459206 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459209 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459211 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459214 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459217 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459220 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459223 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459225 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459228 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459230 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459233 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459235 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459237 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459240 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459242 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459244 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459247 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459250 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:28:38.462446 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459252 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459255 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459258 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459655 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459660 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459663 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459666 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459669 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459672 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459675 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459677 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459680 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459682 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459685 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459688 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459690 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459692 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459695 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459697 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459700 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:28:38.462923 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459702 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459705 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459707 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459709 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459712 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459715 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459717 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459720 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459722 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459725 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459727 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459730 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459732 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459738 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459741 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459744 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459746 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459749 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459751 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459754 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:28:38.463405 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459756 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459759 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459761 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459765 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459769 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459772 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459774 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459777 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459780 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459783 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459786 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459788 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459791 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459794 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459796 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459799 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459820 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459823 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:28:38.463910 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459825 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459828 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459830 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459833 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459837 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459840 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459843 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459847 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459850 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459852 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459855 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459858 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459861 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459863 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459866 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459868 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459871 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459874 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459876 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459879 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:28:38.464379 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459882 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459884 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459887 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459889 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459892 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459894 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459896 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459899 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459901 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459904 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459906 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459909 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459911 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459914 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459916 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459918 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459921 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459923 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459926 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459930 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:28:38.464881 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459932 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459935 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459938 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.459942 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460011 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460020 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460031 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460039 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460046 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460050 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460054 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460058 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460061 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460064 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460067 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460070 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460073 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460076 2573 flags.go:64] FLAG: --cgroup-root="" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460079 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460082 2573 flags.go:64] FLAG: --client-ca-file="" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460085 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460088 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460092 2573 flags.go:64] FLAG: --cluster-domain="" Jul 15 19:28:38.465357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460095 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460098 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460108 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460113 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460117 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460120 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460123 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460126 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460130 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460133 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460136 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460139 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460141 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460146 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460149 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460152 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460155 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460157 2573 flags.go:64] FLAG: --enable-server="true" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460160 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460165 2573 flags.go:64] FLAG: --event-burst="100" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460168 2573 flags.go:64] FLAG: --event-qps="50" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460170 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460173 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460180 2573 flags.go:64] FLAG: --eviction-hard="" Jul 15 19:28:38.465923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460184 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460186 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460189 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460192 2573 flags.go:64] FLAG: --eviction-soft="" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460195 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460198 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460201 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460204 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460207 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460210 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460214 2573 flags.go:64] FLAG: --feature-gates="" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460218 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460221 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460224 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460227 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460230 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460233 2573 flags.go:64] FLAG: --help="false" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460236 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460239 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460242 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460245 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460248 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460252 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460255 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 19:28:38.466484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460258 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460261 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460263 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460266 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460269 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460272 2573 flags.go:64] FLAG: --kube-reserved="" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460275 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460278 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460281 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460284 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460287 2573 flags.go:64] FLAG: --lock-file="" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460290 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460293 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460296 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460301 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460304 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460307 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460310 2573 flags.go:64] FLAG: --logging-format="text" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460313 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460316 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460319 2573 flags.go:64] FLAG: --manifest-url="" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460322 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460326 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460329 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460334 2573 flags.go:64] FLAG: --max-pods="110" Jul 15 19:28:38.467517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460336 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460339 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460342 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460345 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460348 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460351 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460354 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460361 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460364 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460367 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460370 2573 flags.go:64] FLAG: --pod-cidr="" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460373 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460378 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460381 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460385 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460388 2573 flags.go:64] FLAG: --port="10250" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460392 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460394 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a333857b69a5f4c7" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460397 2573 flags.go:64] FLAG: --qos-reserved="" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460400 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460403 2573 flags.go:64] FLAG: --register-node="true" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460406 2573 flags.go:64] FLAG: --register-with-taints="" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460410 2573 flags.go:64] FLAG: --registry-burst="10" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460413 2573 flags.go:64] FLAG: --registry-qps="5" Jul 15 19:28:38.468623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460416 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460418 2573 flags.go:64] FLAG: --reserved-memory="" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460422 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460425 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460428 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460431 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460434 2573 flags.go:64] FLAG: --runonce="false" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460437 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460440 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460443 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460446 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460449 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460452 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460455 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460459 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460462 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460465 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460467 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460470 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460473 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460476 2573 flags.go:64] FLAG: --system-cgroups="" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460479 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460485 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460487 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460491 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 19:28:38.469363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460495 2573 flags.go:64] FLAG: --tls-min-version="" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460497 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460500 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460502 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460505 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460508 2573 flags.go:64] FLAG: --v="2" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460512 2573 flags.go:64] FLAG: --version="false" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460517 2573 flags.go:64] FLAG: --vmodule="" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460521 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460524 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460647 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460650 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460653 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460656 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460659 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460662 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460665 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460668 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460670 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460673 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460677 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:28:38.470157 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460681 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460685 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460689 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460692 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460695 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460698 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460701 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460703 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460707 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460710 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460712 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460715 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460717 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460721 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460724 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460726 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460729 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460732 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460735 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460739 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:28:38.471079 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460741 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460744 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460746 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460749 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460752 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460754 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460757 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460759 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460762 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460764 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460767 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460770 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460772 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460774 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460777 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460780 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460782 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460785 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460787 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460790 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:28:38.471671 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460793 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460796 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460798 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460815 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460818 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460820 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460822 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460825 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460827 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460830 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460834 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460836 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460839 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460842 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460844 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460847 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460849 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460851 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460854 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460856 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:28:38.472425 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460859 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460861 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460864 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460866 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460869 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460871 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460874 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460876 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460879 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460882 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460884 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460887 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460890 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460893 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460895 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460898 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460901 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460903 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460905 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:28:38.473254 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460908 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460910 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460913 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460917 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460919 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460922 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460924 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460926 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.460929 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.460935 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.468717 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.468741 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.469839 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.469854 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469942 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:28:38.473768 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469953 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469958 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469962 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469966 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469970 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469975 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469979 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469983 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469987 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469991 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.469996 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470000 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470004 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470008 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470012 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470017 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470021 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470025 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470029 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:28:38.474282 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470033 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470037 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470041 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470045 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470049 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470053 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470057 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470061 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470065 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470070 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470075 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470079 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470083 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470087 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470092 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470097 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470100 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470104 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470108 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470112 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:28:38.474772 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470116 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470120 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470124 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470128 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470132 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470139 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470146 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470150 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470157 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470162 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470166 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470170 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470174 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470178 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470182 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470186 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470191 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470195 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470199 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:28:38.475348 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470203 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470207 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470212 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470216 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470221 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470225 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470229 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470233 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470238 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470242 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470246 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470250 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470254 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470258 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470263 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470266 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470270 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470274 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470278 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470282 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:28:38.475918 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470286 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470290 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470294 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470298 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470302 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470306 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470310 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470314 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470318 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470322 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470326 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470331 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470335 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470339 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470344 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470348 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470352 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470356 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470362 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:28:38.476421 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470368 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.470376 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470556 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470564 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470569 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470573 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470577 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470581 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470585 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470589 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470593 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470597 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470602 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470606 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470609 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:28:38.476968 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470613 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470617 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470621 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470625 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470630 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470634 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470638 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470642 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470648 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470653 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470658 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470662 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470666 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470670 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470674 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470678 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470682 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470686 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470691 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:28:38.477364 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470695 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470700 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470704 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470709 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470712 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470716 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470721 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470725 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470729 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470733 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470737 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470741 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470745 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470749 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470753 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470757 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470761 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470765 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470769 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470773 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:28:38.477844 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470778 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470781 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470786 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470791 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470795 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470799 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470825 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470829 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470833 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470837 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470840 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470844 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470848 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470853 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470862 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470866 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470870 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470875 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470879 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470882 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:28:38.478502 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470886 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470891 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470895 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470899 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470903 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470907 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470911 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470915 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470919 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470923 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470930 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470935 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470939 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470944 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470949 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470953 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470958 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470962 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470966 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:28:38.479173 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470971 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470975 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470979 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470984 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470988 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470992 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.470999 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:38.471004 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.471013 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.471164 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.474024 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.475608 2573 server.go:1019] "Starting client certificate rotation" Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.475722 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:28:38.479876 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.475763 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:28:38.480465 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.480441 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:28:38.482090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.482074 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:28:38.493324 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.493307 2573 log.go:25] "Validated CRI v1 runtime API" Jul 15 19:28:38.496731 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.496717 2573 log.go:25] "Validated CRI v1 image API" Jul 15 19:28:38.498379 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.498365 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 19:28:38.500996 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.500970 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c123f623-6e4f-4c9d-b4cf-57653feaafe0:/dev/nvme0n1p3 f0d125fb-0ddf-4fd4-8ddc-0067a6b67106:/dev/nvme0n1p4] Jul 15 19:28:38.501084 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.500994 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 19:28:38.501260 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.501241 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:28:38.507145 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.507029 2573 manager.go:216] Machine: {Timestamp:2026-07-15 19:28:38.505692957 +0000 UTC m=+0.366897313 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108664 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec289f66ed69f095f5463f948daed24c SystemUUID:ec289f66-ed69-f095-f546-3f948daed24c BootID:1bb07cd9-9760-48cc-ba86-b664ab2693d3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:9c:c1:7d:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:9c:c1:7d:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:1f:2b:7b:0e:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 19:28:38.507145 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.507136 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 19:28:38.507299 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.507279 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 19:28:38.510646 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.510624 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 19:28:38.510778 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.510649 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 19:28:38.512839 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.512828 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 19:28:38.512873 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.512842 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 19:28:38.512873 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.512849 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:28:38.513635 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.513625 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:28:38.514733 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.514724 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 19:28:38.518621 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.518610 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:28:38.518731 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.518721 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 19:28:38.518773 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.518749 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 15 19:28:38.518773 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.518759 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 19:28:38.518773 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.518773 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 19:28:38.518900 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.518785 2573 kubelet.go:398] "Adding apiserver pod source" Jul 15 19:28:38.518900 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.518795 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 19:28:38.519116 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.519102 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d6pk5" Jul 15 19:28:38.519197 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.519185 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:28:38.519232 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.519207 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:28:38.525279 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.525260 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 19:28:38.526821 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.526789 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d6pk5" Jul 15 19:28:38.526965 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.526952 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 19:28:38.527009 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.526982 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 19:28:38.527720 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527708 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527723 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527729 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527734 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527742 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527747 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527781 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527786 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 19:28:38.527792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527793 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 19:28:38.528050 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527799 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 19:28:38.528050 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527827 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 19:28:38.528050 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.527837 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 19:28:38.528497 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.528475 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 19:28:38.528551 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.528476 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 19:28:38.528729 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.528717 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 19:28:38.528729 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.528729 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 19:28:38.531391 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.531373 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 19:28:38.532319 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.532287 2573 server.go:1268] "Started kubelet" Jul 15 19:28:38.532595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.532318 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 19:28:38.532595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.532403 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 19:28:38.532595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.532473 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 19:28:38.533143 ip-10-0-142-40 systemd[1]: Started Kubernetes Kubelet. Jul 15 19:28:38.533987 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.533879 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 19:28:38.534857 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.534840 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 15 19:28:38.539759 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.539734 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 19:28:38.540735 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.540720 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 19:28:38.540795 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.540780 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 19:28:38.540862 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.540819 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:38.541547 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541522 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 19:28:38.541547 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541524 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 19:28:38.541547 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541547 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 19:28:38.541731 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541602 2573 factory.go:55] Registering systemd factory Jul 15 19:28:38.541731 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541617 2573 factory.go:223] Registration of the systemd container factory successfully Jul 15 19:28:38.541731 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541667 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 15 19:28:38.541731 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541676 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 15 19:28:38.542060 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.541873 2573 factory.go:153] Registering CRI-O factory Jul 15 19:28:38.542128 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.542065 2573 factory.go:223] Registration of the crio container factory successfully Jul 15 19:28:38.542183 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.542155 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 19:28:38.542183 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.542182 2573 factory.go:103] Registering Raw factory Jul 15 19:28:38.542277 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.542197 2573 manager.go:1196] Started watching for new ooms in manager Jul 15 19:28:38.542528 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.542514 2573 manager.go:349] Starting recovery of all containers Jul 15 19:28:38.543694 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.543628 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:28:38.546610 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.546583 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-40.ec2.internal\" not found" node="ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.554107 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.554088 2573 manager.go:354] Recovery completed Jul 15 19:28:38.555496 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.555467 2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 15 19:28:38.558455 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.558436 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:28:38.560384 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.560368 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:28:38.560451 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.560399 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:28:38.560451 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.560413 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:28:38.560969 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.560954 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 19:28:38.560969 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.560966 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:28:38.561050 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.560970 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 19:28:38.561050 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.560987 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:28:38.561599 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.561587 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:28:38.561641 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.561612 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:28:38.561641 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.561623 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:28:38.562080 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.562069 2573 policy_none.go:49] "None policy: Start" Jul 15 19:28:38.562120 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.562085 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 19:28:38.562120 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.562095 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 19:28:38.563199 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.563188 2573 policy_none.go:47] "Start" Jul 15 19:28:38.594698 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.594680 2573 manager.go:337] "Starting Device Plugin manager" Jul 15 19:28:38.594792 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.594717 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 19:28:38.594792 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.594730 2573 server.go:87] "Starting device plugin registration server" Jul 15 19:28:38.594991 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.594980 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 19:28:38.595036 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.594993 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 19:28:38.595129 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.595110 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 19:28:38.595212 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.595200 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 19:28:38.595248 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.595213 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 19:28:38.595891 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.595874 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 19:28:38.595955 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.595915 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:38.687503 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.687447 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 19:28:38.688874 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.688858 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 19:28:38.688937 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.688898 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 19:28:38.688937 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.688924 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 19:28:38.689019 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.688958 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 19:28:38.691877 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.691849 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:28:38.695176 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.695162 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:28:38.696017 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.696004 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:28:38.696078 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.696031 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:28:38.696078 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.696042 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:28:38.696078 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.696063 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.707458 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.707412 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.707458 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.707433 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-40.ec2.internal\": node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:38.724435 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.724414 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:38.789745 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.789704 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal"] Jul 15 19:28:38.789877 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.789815 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:28:38.791822 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.791796 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:28:38.791905 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.791837 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:28:38.791905 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.791847 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:28:38.793166 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.793152 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:28:38.793302 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.793286 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.793362 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.793328 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:28:38.794274 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.794260 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:28:38.794336 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.794287 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:28:38.794336 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.794301 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:28:38.794634 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.794620 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:28:38.794686 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.794645 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:28:38.794686 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.794657 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:28:38.795871 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.795858 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.795920 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.795881 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:28:38.797632 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.797615 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:28:38.797729 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.797642 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:28:38.797729 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.797657 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:28:38.816395 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.816378 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-40.ec2.internal\" not found" node="ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.820705 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.820690 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-40.ec2.internal\" not found" node="ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.824751 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.824736 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:38.843005 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.842983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0367d4aa1f49474b3ec99a97b84a097-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal\" (UID: \"f0367d4aa1f49474b3ec99a97b84a097\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.843080 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.843008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb0fb17f4c4321230ae8f9ba72a0414e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-40.ec2.internal\" (UID: \"eb0fb17f4c4321230ae8f9ba72a0414e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.843080 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.843027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0367d4aa1f49474b3ec99a97b84a097-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal\" (UID: \"f0367d4aa1f49474b3ec99a97b84a097\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.925621 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:38.925588 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:38.944008 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.943972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb0fb17f4c4321230ae8f9ba72a0414e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-40.ec2.internal\" (UID: \"eb0fb17f4c4321230ae8f9ba72a0414e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.944008 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.944008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0367d4aa1f49474b3ec99a97b84a097-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal\" (UID: \"f0367d4aa1f49474b3ec99a97b84a097\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.944209 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.944036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0367d4aa1f49474b3ec99a97b84a097-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal\" (UID: \"f0367d4aa1f49474b3ec99a97b84a097\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.944209 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.944078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0367d4aa1f49474b3ec99a97b84a097-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal\" (UID: \"f0367d4aa1f49474b3ec99a97b84a097\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.944209 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.944097 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb0fb17f4c4321230ae8f9ba72a0414e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-40.ec2.internal\" (UID: \"eb0fb17f4c4321230ae8f9ba72a0414e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" Jul 15 19:28:38.944209 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:38.944161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0367d4aa1f49474b3ec99a97b84a097-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal\" (UID: \"f0367d4aa1f49474b3ec99a97b84a097\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:39.026453 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.026381 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.118923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.118894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:39.123841 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.123822 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" Jul 15 19:28:39.126531 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.126500 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.227230 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.227186 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.327783 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.327720 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.428271 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.428228 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.474800 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.474771 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 19:28:39.475402 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.474979 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:28:39.475402 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.474979 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:28:39.529365 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.529324 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.531523 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.531483 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 19:23:38 +0000 UTC" deadline="2028-01-23 06:21:56.569189381 +0000 UTC" Jul 15 19:28:39.531523 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.531523 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13354h53m17.037669246s" Jul 15 19:28:39.629517 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.629440 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.730124 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.730078 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.770999 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.770967 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:28:39.830589 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.830557 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:39.894318 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:39.894277 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0fb17f4c4321230ae8f9ba72a0414e.slice/crio-05fe23bc8eddeea597bfbd543bbe4448a3e1ae2b0783e0d34a35b85f78d9d640 WatchSource:0}: Error finding container 05fe23bc8eddeea597bfbd543bbe4448a3e1ae2b0783e0d34a35b85f78d9d640: Status 404 returned error can't find the container with id 05fe23bc8eddeea597bfbd543bbe4448a3e1ae2b0783e0d34a35b85f78d9d640 Jul 15 19:28:39.894650 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:39.894630 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0367d4aa1f49474b3ec99a97b84a097.slice/crio-796aea5f055c29fd3435162ab60fb04a67c342e0a970c4d5fe1c3d52b153e027 WatchSource:0}: Error finding container 796aea5f055c29fd3435162ab60fb04a67c342e0a970c4d5fe1c3d52b153e027: Status 404 returned error can't find the container with id 796aea5f055c29fd3435162ab60fb04a67c342e0a970c4d5fe1c3d52b153e027 Jul 15 19:28:39.900702 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:39.900682 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:28:39.931438 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:39.931402 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:40.032248 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.032197 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-40.ec2.internal\" not found" Jul 15 19:28:40.115127 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.115099 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:28:40.141728 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.141660 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" Jul 15 19:28:40.148358 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.148335 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:28:40.149915 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.149902 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" Jul 15 19:28:40.159398 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.159383 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:28:40.299902 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.299866 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:28:40.519335 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.519306 2573 apiserver.go:52] "Watching apiserver" Jul 15 19:28:40.525492 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.525468 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 19:28:40.528033 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.528004 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-6qp4s","kube-system/global-pull-secret-syncer-f9fvm","kube-system/konnectivity-agent-x5b2v","openshift-dns/node-resolver-7l44b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal","openshift-multus/multus-tlph8","openshift-multus/network-metrics-daemon-j5zpl","openshift-network-operator/iptables-alerter-q8lq9","kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h","openshift-cluster-node-tuning-operator/tuned-r7mkh","openshift-image-registry/node-ca-mzq5g","openshift-multus/multus-additional-cni-plugins-gfr4h","openshift-network-diagnostics/network-check-target-lwlwl"] Jul 15 19:28:40.529554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.529531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:40.529652 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.529628 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:40.530711 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.530688 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.531871 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.531848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.533023 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.533001 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:28:40.533130 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.533055 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 19:28:40.533202 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.533185 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4pxk6\"" Jul 15 19:28:40.533259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.533218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.533848 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.533827 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 19:28:40.534742 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.534711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:40.534828 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.534807 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:40.534989 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.534972 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 19:28:40.535125 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.535106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xxc5f\"" Jul 15 19:28:40.536377 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.535701 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 19:28:40.536377 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.535902 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkvtw\"" Jul 15 19:28:40.536377 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.536205 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 19:28:40.537877 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.537837 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.539658 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.539635 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.539782 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.539763 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 19:28:40.540063 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.540041 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 19:28:40.540358 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.540298 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 19:28:40.540358 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.540316 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 19:28:40.540358 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.540334 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-476jv\"" Jul 15 19:28:40.541357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.541336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.541803 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.541784 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 19:28:40.542109 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.542089 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:28:40.542109 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.542103 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-psgg2\"" Jul 15 19:28:40.542256 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.542149 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 19:28:40.542424 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.542405 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 19:28:40.542588 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.542571 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.542672 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.542631 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:40.543548 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.543519 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 19:28:40.543651 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.543605 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 19:28:40.543912 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.543856 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q9sqs\"" Jul 15 19:28:40.544179 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.544161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.545529 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.545513 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.546357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.546334 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 19:28:40.546486 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.546375 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 19:28:40.546579 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.546560 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z7xb4\"" Jul 15 19:28:40.546661 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.546626 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 19:28:40.546826 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.546792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.547764 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.547716 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 19:28:40.547764 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.547723 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 19:28:40.548236 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.548214 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 19:28:40.548374 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.548351 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 19:28:40.548454 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.548279 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 19:28:40.548517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.548241 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d5vr5\"" Jul 15 19:28:40.548576 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.548294 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 19:28:40.548832 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.548814 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 19:28:40.549120 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.549095 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 19:28:40.549120 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.549113 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 19:28:40.549268 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.549100 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-69474\"" Jul 15 19:28:40.553397 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-netns\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.553499 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-conf-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.553499 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553432 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-system-cni-dir\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.553624 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.553624 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.553624 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-tmp\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.553624 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553589 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f36152a-ffe5-4e8d-b335-d52517232ba1-konnectivity-ca\") pod \"konnectivity-agent-x5b2v\" (UID: \"0f36152a-ffe5-4e8d-b335-d52517232ba1\") " pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-system-cni-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-cnibin\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-os-release\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-cni-bin\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553763 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysconfig\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysctl-conf\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-tuned\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.553829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-socket-dir-parent\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cnibin\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.553964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-kubernetes\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-host\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fp2\" (UniqueName: \"kubernetes.io/projected/2514b799-dd92-4c5d-9a7e-63f79cab71ef-kube-api-access-29fp2\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-iptables-alerter-script\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-host-slash\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-os-release\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df75c58a-3f8a-4b70-8a59-a768bb296b53-kubelet-config\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-run\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-lib-modules\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f36152a-ffe5-4e8d-b335-d52517232ba1-agent-certs\") pod \"konnectivity-agent-x5b2v\" (UID: \"0f36152a-ffe5-4e8d-b335-d52517232ba1\") " pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-cni-multus\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.554342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-daemon-config\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-multus-certs\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df75c58a-3f8a-4b70-8a59-a768bb296b53-dbus\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-cni-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554424 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-hostroot\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d79v\" (UniqueName: \"kubernetes.io/projected/00ec9d85-c523-457e-8688-a96deb7d1ed7-kube-api-access-5d79v\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxc2n\" (UniqueName: \"kubernetes.io/projected/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-kube-api-access-wxc2n\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7mc\" (UniqueName: \"kubernetes.io/projected/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-kube-api-access-zz7mc\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554585 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysctl-d\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-systemd\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-kube-api-access-9xmtd\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2514b799-dd92-4c5d-9a7e-63f79cab71ef-tmp-dir\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00ec9d85-c523-457e-8688-a96deb7d1ed7-cni-binary-copy\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gctb\" (UniqueName: \"kubernetes.io/projected/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-kube-api-access-4gctb\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554714 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-modprobe-d\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.555053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-var-lib-kubelet\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.555842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2514b799-dd92-4c5d-9a7e-63f79cab71ef-hosts-file\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.555842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:40.555842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-kubelet\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-etc-kubernetes\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.555842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.555842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-sys\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.555842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.554959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-k8s-cni-cncf-io\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.582543 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.581781 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p2s4b" Jul 15 19:28:40.589381 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.589362 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p2s4b" Jul 15 19:28:40.643023 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.643001 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 19:28:40.655883 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.655845 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-var-lib-kubelet\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.655995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.655895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.655995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.655923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.655995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.655942 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-sys\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.655995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.655958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-conf-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.655995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.655983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-systemd-units\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-ovn\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656018 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-var-lib-kubelet\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656059 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-sys\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656070 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-conf-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-env-overrides\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-tmp\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-system-cni-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656280 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-system-cni-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.656310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-cnibin\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-cnibin\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-os-release\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc2dcea3-f197-4b33-b77f-00716c3c698c-serviceca\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-os-release\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97qw\" (UniqueName: \"kubernetes.io/projected/cc2dcea3-f197-4b33-b77f-00716c3c698c-kube-api-access-m97qw\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656424 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj4h2\" (UniqueName: \"kubernetes.io/projected/5f82c026-8052-4285-899a-5df9678180a1-kube-api-access-fj4h2\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysconfig\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656462 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysctl-conf\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysconfig\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cnibin\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656635 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysctl-conf\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-slash\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-var-lib-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656637 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cnibin\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.656716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-kubernetes\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-host\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656806 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-host\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-kubernetes\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-host-slash\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df75c58a-3f8a-4b70-8a59-a768bb296b53-kubelet-config\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-run-netns\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656954 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/df75c58a-3f8a-4b70-8a59-a768bb296b53-kubelet-config\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-host-slash\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.656980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovnkube-config\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-multus-certs\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df75c58a-3f8a-4b70-8a59-a768bb296b53-dbus\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-multus-certs\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-etc-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovn-node-metrics-cert\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-cni-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.657554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-hostroot\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657161 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/df75c58a-3f8a-4b70-8a59-a768bb296b53-dbus\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d79v\" (UniqueName: \"kubernetes.io/projected/00ec9d85-c523-457e-8688-a96deb7d1ed7-kube-api-access-5d79v\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657203 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-cni-dir\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657207 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-hostroot\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657246 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7mc\" (UniqueName: \"kubernetes.io/projected/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-kube-api-access-zz7mc\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-systemd\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00ec9d85-c523-457e-8688-a96deb7d1ed7-cni-binary-copy\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc2dcea3-f197-4b33-b77f-00716c3c698c-host\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-systemd\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-sys-fs\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-systemd\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovnkube-script-lib\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-modprobe-d\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2514b799-dd92-4c5d-9a7e-63f79cab71ef-hosts-file\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:40.658433 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-kubelet\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-modprobe-d\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-etc-kubernetes\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2514b799-dd92-4c5d-9a7e-63f79cab71ef-hosts-file\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657690 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-etc-kubernetes\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-k8s-cni-cncf-io\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-k8s-cni-cncf-io\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657727 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-kubelet\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-netns\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657811 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-run-netns\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-system-cni-dir\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657879 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-system-cni-dir\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657920 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00ec9d85-c523-457e-8688-a96deb7d1ed7-cni-binary-copy\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.657980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-kubelet\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f36152a-ffe5-4e8d-b335-d52517232ba1-konnectivity-ca\") pod \"konnectivity-agent-x5b2v\" (UID: \"0f36152a-ffe5-4e8d-b335-d52517232ba1\") " pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.659252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-cni-bin\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-registration-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-cni-bin\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpq9\" (UniqueName: \"kubernetes.io/projected/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-kube-api-access-mtpq9\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-tuned\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.658197 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-socket-dir-parent\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-socket-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658294 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-device-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fp2\" (UniqueName: \"kubernetes.io/projected/2514b799-dd92-4c5d-9a7e-63f79cab71ef-kube-api-access-29fp2\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658331 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-iptables-alerter-script\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658410 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-socket-dir-parent\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.658552 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:41.158528555 +0000 UTC m=+3.019732905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f36152a-ffe5-4e8d-b335-d52517232ba1-konnectivity-ca\") pod \"konnectivity-agent-x5b2v\" (UID: \"0f36152a-ffe5-4e8d-b335-d52517232ba1\") " pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.660314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658571 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-os-release\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-log-socket\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-cni-bin\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658726 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-os-release\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-run\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-lib-modules\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-run\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f36152a-ffe5-4e8d-b335-d52517232ba1-agent-certs\") pod \"konnectivity-agent-x5b2v\" (UID: \"0f36152a-ffe5-4e8d-b335-d52517232ba1\") " pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-cni-multus\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658898 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-lib-modules\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-daemon-config\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxc2n\" (UniqueName: \"kubernetes.io/projected/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-kube-api-access-wxc2n\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00ec9d85-c523-457e-8688-a96deb7d1ed7-host-var-lib-cni-multus\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.658972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-node-log\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.661165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-cni-netd\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysctl-d\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-kube-api-access-9xmtd\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2514b799-dd92-4c5d-9a7e-63f79cab71ef-tmp-dir\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.659141 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659087 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-iptables-alerter-script\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.659191 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret podName:df75c58a-3f8a-4b70-8a59-a768bb296b53 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:41.159173597 +0000 UTC m=+3.020377930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret") pod "global-pull-secret-syncer-f9fvm" (UID: "df75c58a-3f8a-4b70-8a59-a768bb296b53") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gctb\" (UniqueName: \"kubernetes.io/projected/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-kube-api-access-4gctb\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659279 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-sysctl-d\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659535 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2514b799-dd92-4c5d-9a7e-63f79cab71ef-tmp-dir\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659803 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00ec9d85-c523-457e-8688-a96deb7d1ed7-multus-daemon-config\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.659977 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-tmp\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.661977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.660796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-etc-tuned\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.662386 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.662193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f36152a-ffe5-4e8d-b335-d52517232ba1-agent-certs\") pod \"konnectivity-agent-x5b2v\" (UID: \"0f36152a-ffe5-4e8d-b335-d52517232ba1\") " pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.664696 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.664672 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:28:40.664804 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.664707 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:28:40.664804 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.664723 2573 projected.go:196] Error preparing data for projected volume kube-api-access-772sg for pod openshift-network-diagnostics/network-check-target-lwlwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:40.664804 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:40.664783 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg podName:3a3888c7-eea2-453e-9cac-805bc3236eb4 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:41.164765622 +0000 UTC m=+3.025969977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-772sg" (UniqueName: "kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg") pod "network-check-target-lwlwl" (UID: "3a3888c7-eea2-453e-9cac-805bc3236eb4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:40.667297 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.667275 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxc2n\" (UniqueName: \"kubernetes.io/projected/f0c390fc-7420-4fd0-9cd6-c9957b144ee6-kube-api-access-wxc2n\") pod \"iptables-alerter-q8lq9\" (UID: \"f0c390fc-7420-4fd0-9cd6-c9957b144ee6\") " pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.667618 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.667587 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d79v\" (UniqueName: \"kubernetes.io/projected/00ec9d85-c523-457e-8688-a96deb7d1ed7-kube-api-access-5d79v\") pod \"multus-tlph8\" (UID: \"00ec9d85-c523-457e-8688-a96deb7d1ed7\") " pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.667713 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.667626 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7mc\" (UniqueName: \"kubernetes.io/projected/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-kube-api-access-zz7mc\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:40.667995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.667967 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fp2\" (UniqueName: \"kubernetes.io/projected/2514b799-dd92-4c5d-9a7e-63f79cab71ef-kube-api-access-29fp2\") pod \"node-resolver-7l44b\" (UID: \"2514b799-dd92-4c5d-9a7e-63f79cab71ef\") " pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.667995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.667957 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmtd\" (UniqueName: \"kubernetes.io/projected/e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b-kube-api-access-9xmtd\") pod \"tuned-r7mkh\" (UID: \"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b\") " pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.668145 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.668107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gctb\" (UniqueName: \"kubernetes.io/projected/48b7ce9d-c732-4b21-a7ef-bd8f9963a183-kube-api-access-4gctb\") pod \"multus-additional-cni-plugins-gfr4h\" (UID: \"48b7ce9d-c732-4b21-a7ef-bd8f9963a183\") " pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.694443 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.694352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" event={"ID":"f0367d4aa1f49474b3ec99a97b84a097","Type":"ContainerStarted","Data":"796aea5f055c29fd3435162ab60fb04a67c342e0a970c4d5fe1c3d52b153e027"} Jul 15 19:28:40.695673 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.695639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" event={"ID":"eb0fb17f4c4321230ae8f9ba72a0414e","Type":"ContainerStarted","Data":"05fe23bc8eddeea597bfbd543bbe4448a3e1ae2b0783e0d34a35b85f78d9d640"} Jul 15 19:28:40.760104 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-kubelet\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-registration-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpq9\" (UniqueName: \"kubernetes.io/projected/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-kube-api-access-mtpq9\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-socket-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-device-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-log-socket\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760210 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-kubelet\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760219 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-registration-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760276 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-device-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760289 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-cni-bin\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-cni-bin\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-log-socket\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-socket-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760383 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760407 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-node-log\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760432 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-cni-netd\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760444 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-node-log\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-cni-netd\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-systemd-units\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760534 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-ovn\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.760595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760575 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-env-overrides\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc2dcea3-f197-4b33-b77f-00716c3c698c-serviceca\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760647 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-systemd-units\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97qw\" (UniqueName: \"kubernetes.io/projected/cc2dcea3-f197-4b33-b77f-00716c3c698c-kube-api-access-m97qw\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj4h2\" (UniqueName: \"kubernetes.io/projected/5f82c026-8052-4285-899a-5df9678180a1-kube-api-access-fj4h2\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-slash\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-var-lib-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-run-netns\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovnkube-config\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-etc-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovn-node-metrics-cert\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760934 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc2dcea3-f197-4b33-b77f-00716c3c698c-host\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-sys-fs\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.760984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-systemd\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.761343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovnkube-script-lib\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761087 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-env-overrides\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761137 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-ovn\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761175 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761359 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc2dcea3-f197-4b33-b77f-00716c3c698c-serviceca\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc2dcea3-f197-4b33-b77f-00716c3c698c-host\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761474 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-etc-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761565 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovnkube-script-lib\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761621 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovnkube-config\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f82c026-8052-4285-899a-5df9678180a1-sys-fs\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761676 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-var-lib-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761718 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-run-netns\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-host-slash\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761677 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-systemd\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.762090 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.761907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-run-openvswitch\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.764610 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.764586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-ovn-node-metrics-cert\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.768414 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.768388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97qw\" (UniqueName: \"kubernetes.io/projected/cc2dcea3-f197-4b33-b77f-00716c3c698c-kube-api-access-m97qw\") pod \"node-ca-mzq5g\" (UID: \"cc2dcea3-f197-4b33-b77f-00716c3c698c\") " pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.768651 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.768630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpq9\" (UniqueName: \"kubernetes.io/projected/61bd4882-80fa-43e2-9dab-ad2dd5ef25d8-kube-api-access-mtpq9\") pod \"ovnkube-node-6qp4s\" (UID: \"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.769257 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.769233 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj4h2\" (UniqueName: \"kubernetes.io/projected/5f82c026-8052-4285-899a-5df9678180a1-kube-api-access-fj4h2\") pod \"aws-ebs-csi-driver-node-sbs7h\" (UID: \"5f82c026-8052-4285-899a-5df9678180a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.845278 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.845245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" Jul 15 19:28:40.853271 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.853237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:28:40.862266 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.862237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7l44b" Jul 15 19:28:40.868154 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.868129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tlph8" Jul 15 19:28:40.877022 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.877003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8lq9" Jul 15 19:28:40.885625 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.885600 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" Jul 15 19:28:40.892326 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.892300 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" Jul 15 19:28:40.902151 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.902129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:28:40.906967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.906949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzq5g" Jul 15 19:28:40.943041 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:40.943015 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:28:41.027210 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:41.027176 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bd4882_80fa_43e2_9dab_ad2dd5ef25d8.slice/crio-eebd24c8e658c57c1064145ba4fcdff1d41b72721d51e40c5ff906f2a7bd5dc2 WatchSource:0}: Error finding container eebd24c8e658c57c1064145ba4fcdff1d41b72721d51e40c5ff906f2a7bd5dc2: Status 404 returned error can't find the container with id eebd24c8e658c57c1064145ba4fcdff1d41b72721d51e40c5ff906f2a7bd5dc2 Jul 15 19:28:41.034921 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:41.034593 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ec9d85_c523_457e_8688_a96deb7d1ed7.slice/crio-5d0a0621aa6d56749b992fdc7ea96ffc552c65b62fe6d3436b4170c926fb2876 WatchSource:0}: Error finding container 5d0a0621aa6d56749b992fdc7ea96ffc552c65b62fe6d3436b4170c926fb2876: Status 404 returned error can't find the container with id 5d0a0621aa6d56749b992fdc7ea96ffc552c65b62fe6d3436b4170c926fb2876 Jul 15 19:28:41.035210 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:41.035181 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f36152a_ffe5_4e8d_b335_d52517232ba1.slice/crio-91446670f25d88141cd33a61971a9981cce58e427fc62adcf301a8e2c6408b76 WatchSource:0}: Error finding container 91446670f25d88141cd33a61971a9981cce58e427fc62adcf301a8e2c6408b76: Status 404 returned error can't find the container with id 91446670f25d88141cd33a61971a9981cce58e427fc62adcf301a8e2c6408b76 Jul 15 19:28:41.036196 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:41.036167 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2514b799_dd92_4c5d_9a7e_63f79cab71ef.slice/crio-1202b9019fd44f88c8e5b8bb63a4aea0775d8ba7ae519f6c524eadb7236b191b WatchSource:0}: Error finding container 1202b9019fd44f88c8e5b8bb63a4aea0775d8ba7ae519f6c524eadb7236b191b: Status 404 returned error can't find the container with id 1202b9019fd44f88c8e5b8bb63a4aea0775d8ba7ae519f6c524eadb7236b191b Jul 15 19:28:41.037389 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:28:41.037298 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e7bed6_c4e2_4ab1_bf98_f9b398a89b9b.slice/crio-90687fecaf1a285506f52a495d25d1661a97cb855b898534d9e35be30e1a103a WatchSource:0}: Error finding container 90687fecaf1a285506f52a495d25d1661a97cb855b898534d9e35be30e1a103a: Status 404 returned error can't find the container with id 90687fecaf1a285506f52a495d25d1661a97cb855b898534d9e35be30e1a103a Jul 15 19:28:41.164665 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.164633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:41.164665 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.164681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:41.164917 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.164794 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:41.164917 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.164800 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:41.164917 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.164870 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret podName:df75c58a-3f8a-4b70-8a59-a768bb296b53 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:42.164855363 +0000 UTC m=+4.026059695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret") pod "global-pull-secret-syncer-f9fvm" (UID: "df75c58a-3f8a-4b70-8a59-a768bb296b53") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:41.164917 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.164884 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:42.164878272 +0000 UTC m=+4.026082604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:41.265521 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.265489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:41.265649 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.265560 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:28:41.265649 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.265582 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:28:41.265649 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.265592 2573 projected.go:196] Error preparing data for projected volume kube-api-access-772sg for pod openshift-network-diagnostics/network-check-target-lwlwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:41.265649 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.265639 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg podName:3a3888c7-eea2-453e-9cac-805bc3236eb4 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:42.265626118 +0000 UTC m=+4.126830451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-772sg" (UniqueName: "kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg") pod "network-check-target-lwlwl" (UID: "3a3888c7-eea2-453e-9cac-805bc3236eb4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:41.591894 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.591784 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:23:40 +0000 UTC" deadline="2027-12-19 05:24:28.710007864 +0000 UTC" Jul 15 19:28:41.591894 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.591841 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12513h55m47.118171963s" Jul 15 19:28:41.690836 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.690115 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:41.690836 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.690264 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:41.690836 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.690687 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:41.690836 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:41.690799 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:41.704360 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.704323 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" event={"ID":"5f82c026-8052-4285-899a-5df9678180a1","Type":"ContainerStarted","Data":"2d138991657255b0c297027f1afb9233f1dea127eef972f405101d9ca518ec35"} Jul 15 19:28:41.712325 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.712292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" event={"ID":"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b","Type":"ContainerStarted","Data":"90687fecaf1a285506f52a495d25d1661a97cb855b898534d9e35be30e1a103a"} Jul 15 19:28:41.718707 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.718675 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5b2v" event={"ID":"0f36152a-ffe5-4e8d-b335-d52517232ba1","Type":"ContainerStarted","Data":"91446670f25d88141cd33a61971a9981cce58e427fc62adcf301a8e2c6408b76"} Jul 15 19:28:41.720726 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.720644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerStarted","Data":"ed4d8becbc9b718e8a2da00d55e59443190f0840e0e50168dee420700d9e5e4d"} Jul 15 19:28:41.726683 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.726657 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"eebd24c8e658c57c1064145ba4fcdff1d41b72721d51e40c5ff906f2a7bd5dc2"} Jul 15 19:28:41.732994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.732949 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzq5g" event={"ID":"cc2dcea3-f197-4b33-b77f-00716c3c698c","Type":"ContainerStarted","Data":"8a0aebe62a5c0419b2283b7f41bde610db39c95a61d0145ffdda09f4dc143823"} Jul 15 19:28:41.737202 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.737150 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8lq9" event={"ID":"f0c390fc-7420-4fd0-9cd6-c9957b144ee6","Type":"ContainerStarted","Data":"efffce637ba3bfdbb80f5239fc5334af3194ca432c31a5d732b9ea82041b4948"} Jul 15 19:28:41.740233 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.740204 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tlph8" event={"ID":"00ec9d85-c523-457e-8688-a96deb7d1ed7","Type":"ContainerStarted","Data":"5d0a0621aa6d56749b992fdc7ea96ffc552c65b62fe6d3436b4170c926fb2876"} Jul 15 19:28:41.742493 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.742463 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7l44b" event={"ID":"2514b799-dd92-4c5d-9a7e-63f79cab71ef","Type":"ContainerStarted","Data":"1202b9019fd44f88c8e5b8bb63a4aea0775d8ba7ae519f6c524eadb7236b191b"} Jul 15 19:28:41.747479 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.747443 2573 generic.go:356] "Generic (PLEG): container finished" podID="f0367d4aa1f49474b3ec99a97b84a097" containerID="2e911f04cc2f415694ab1a670e3ad6e6ee6b7311fed7e675de1e5c6e73dd1f92" exitCode=0 Jul 15 19:28:41.747595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:41.747494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" event={"ID":"f0367d4aa1f49474b3ec99a97b84a097","Type":"ContainerDied","Data":"2e911f04cc2f415694ab1a670e3ad6e6ee6b7311fed7e675de1e5c6e73dd1f92"} Jul 15 19:28:42.174718 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.173863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:42.174718 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.173926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:42.174718 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.174085 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:42.174718 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.174158 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret podName:df75c58a-3f8a-4b70-8a59-a768bb296b53 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:44.174135466 +0000 UTC m=+6.035339807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret") pod "global-pull-secret-syncer-f9fvm" (UID: "df75c58a-3f8a-4b70-8a59-a768bb296b53") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:42.174718 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.174611 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:42.174718 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.174672 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:44.174653057 +0000 UTC m=+6.035857392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:42.274610 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.274567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:42.274821 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.274800 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:28:42.274926 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.274826 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:28:42.274926 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.274839 2573 projected.go:196] Error preparing data for projected volume kube-api-access-772sg for pod openshift-network-diagnostics/network-check-target-lwlwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:42.274926 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.274925 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg podName:3a3888c7-eea2-453e-9cac-805bc3236eb4 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:44.274905807 +0000 UTC m=+6.136110159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-772sg" (UniqueName: "kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg") pod "network-check-target-lwlwl" (UID: "3a3888c7-eea2-453e-9cac-805bc3236eb4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:42.592855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.592710 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:23:40 +0000 UTC" deadline="2028-01-29 16:23:03.442329582 +0000 UTC" Jul 15 19:28:42.592855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.592780 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13508h54m20.849554459s" Jul 15 19:28:42.692642 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.692607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:42.692837 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:42.692748 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:42.755707 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.755664 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" event={"ID":"f0367d4aa1f49474b3ec99a97b84a097","Type":"ContainerStarted","Data":"264d134c1b505c8feb58a89a7e6b239abff30878d0d905776daa8cf6e621b7a7"} Jul 15 19:28:42.770182 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:42.770111 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-40.ec2.internal" podStartSLOduration=2.770088254 podStartE2EDuration="2.770088254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:28:42.769459238 +0000 UTC m=+4.630663596" watchObservedRunningTime="2026-07-15 19:28:42.770088254 +0000 UTC m=+4.631292602" Jul 15 19:28:43.689697 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:43.689640 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:43.690225 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:43.689874 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:43.691063 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:43.690306 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:43.691063 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:43.690404 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:44.192708 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:44.192667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:44.192953 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:44.192732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:44.192953 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.192887 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:44.192953 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.192952 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret podName:df75c58a-3f8a-4b70-8a59-a768bb296b53 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:48.192932383 +0000 UTC m=+10.054136721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret") pod "global-pull-secret-syncer-f9fvm" (UID: "df75c58a-3f8a-4b70-8a59-a768bb296b53") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:44.193416 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.193395 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:44.193520 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.193488 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:48.193469638 +0000 UTC m=+10.054673979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:44.294955 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:44.294154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:44.294955 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.294418 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:28:44.294955 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.294444 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:28:44.294955 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.294458 2573 projected.go:196] Error preparing data for projected volume kube-api-access-772sg for pod openshift-network-diagnostics/network-check-target-lwlwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:44.294955 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.294525 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg podName:3a3888c7-eea2-453e-9cac-805bc3236eb4 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:48.294505024 +0000 UTC m=+10.155709360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-772sg" (UniqueName: "kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg") pod "network-check-target-lwlwl" (UID: "3a3888c7-eea2-453e-9cac-805bc3236eb4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:44.689986 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:44.689949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:44.690556 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:44.690075 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:45.689911 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:45.689876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:45.690097 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:45.690024 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:45.690097 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:45.690086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:45.690500 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:45.690151 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:46.691580 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:46.691542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:46.692063 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:46.691672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:47.690399 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:47.689924 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:47.690399 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:47.689980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:47.690399 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:47.690075 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:47.690399 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:47.690207 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:48.225173 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:48.225140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:48.225583 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:48.225193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:48.225583 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.225327 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:48.225583 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.225382 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret podName:df75c58a-3f8a-4b70-8a59-a768bb296b53 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:56.225364287 +0000 UTC m=+18.086568622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret") pod "global-pull-secret-syncer-f9fvm" (UID: "df75c58a-3f8a-4b70-8a59-a768bb296b53") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:48.225770 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.225751 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:48.225866 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.225828 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:56.225793206 +0000 UTC m=+18.086997540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:48.325798 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:48.325761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:48.326055 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.326032 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:28:48.326145 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.326061 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:28:48.326145 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.326075 2573 projected.go:196] Error preparing data for projected volume kube-api-access-772sg for pod openshift-network-diagnostics/network-check-target-lwlwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:48.326219 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.326165 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg podName:3a3888c7-eea2-453e-9cac-805bc3236eb4 nodeName:}" failed. No retries permitted until 2026-07-15 19:28:56.326146481 +0000 UTC m=+18.187350815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-772sg" (UniqueName: "kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg") pod "network-check-target-lwlwl" (UID: "3a3888c7-eea2-453e-9cac-805bc3236eb4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:48.691676 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:48.691644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:48.691865 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:48.691740 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:49.689151 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:49.689117 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:49.689565 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:49.689256 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:49.689565 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:49.689276 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:49.689565 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:49.689350 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:50.689429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:50.689399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:50.689836 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:50.689511 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:51.689440 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:51.689406 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:51.689851 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:51.689530 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:51.689851 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:51.689594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:51.689851 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:51.689700 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:52.689690 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:52.689658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:52.690169 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:52.689825 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:53.689894 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:53.689861 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:53.690312 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:53.689861 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:53.690312 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:53.689991 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:53.690312 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:53.690096 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:54.689273 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:54.689240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:54.689435 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:54.689353 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:55.689145 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:55.689109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:55.689634 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:55.689109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:55.689634 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:55.689228 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:55.689634 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:55.689289 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:56.286888 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:56.286852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:56.287144 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:56.286898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:56.287144 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.287000 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:56.287144 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.287007 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:56.287144 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.287059 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret podName:df75c58a-3f8a-4b70-8a59-a768bb296b53 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:12.28704416 +0000 UTC m=+34.148248496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret") pod "global-pull-secret-syncer-f9fvm" (UID: "df75c58a-3f8a-4b70-8a59-a768bb296b53") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:28:56.287144 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.287076 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:12.287067484 +0000 UTC m=+34.148271814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:28:56.388004 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:56.387970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:56.388182 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.388149 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:28:56.388182 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.388176 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:28:56.388293 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.388189 2573 projected.go:196] Error preparing data for projected volume kube-api-access-772sg for pod openshift-network-diagnostics/network-check-target-lwlwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:56.388293 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.388259 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg podName:3a3888c7-eea2-453e-9cac-805bc3236eb4 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:12.388240821 +0000 UTC m=+34.249445153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-772sg" (UniqueName: "kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg") pod "network-check-target-lwlwl" (UID: "3a3888c7-eea2-453e-9cac-805bc3236eb4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:28:56.689177 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:56.689138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:56.689639 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:56.689278 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:57.689268 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:57.689232 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:57.689598 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:57.689272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:57.689598 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:57.689345 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:57.689598 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:57.689404 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:28:58.690833 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:58.690783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:28:58.691257 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:58.690937 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:28:59.689879 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:59.689844 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:28:59.689879 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:28:59.689878 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:28:59.690094 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:59.690003 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:28:59.690142 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:28:59.690091 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:00.689595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:00.689560 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:00.690081 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:00.689708 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:29:01.689938 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.689677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:01.690652 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.689698 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:01.690652 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:01.690072 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:29:01.690652 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:01.690181 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:01.804229 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.804199 2573 generic.go:356] "Generic (PLEG): container finished" podID="48b7ce9d-c732-4b21-a7ef-bd8f9963a183" containerID="e5c1528874ee442e936609b708e90b5435bd1f13dbd4d477365aeffa37ce3792" exitCode=0 Jul 15 19:29:01.804375 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.804271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerDied","Data":"e5c1528874ee442e936609b708e90b5435bd1f13dbd4d477365aeffa37ce3792"} Jul 15 19:29:01.805940 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.805921 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:29:01.806295 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.806272 2573 generic.go:356] "Generic (PLEG): container finished" podID="61bd4882-80fa-43e2-9dab-ad2dd5ef25d8" containerID="e892b501da48d76e047e0aab165fd1a9a0701684e6671477f9fa105901f83e80" exitCode=1 Jul 15 19:29:01.806381 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.806341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerDied","Data":"e892b501da48d76e047e0aab165fd1a9a0701684e6671477f9fa105901f83e80"} Jul 15 19:29:01.806432 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.806405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"314c873f8a5a3284a4bcd87032f75d59386a39fd6e5aeafb8ea5b66ae150b56e"} Jul 15 19:29:01.807927 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.807905 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzq5g" event={"ID":"cc2dcea3-f197-4b33-b77f-00716c3c698c","Type":"ContainerStarted","Data":"1035ba9ef6853e4dfc0a61c2b000de7b628f59c6fcb88530437141172f52a5ac"} Jul 15 19:29:01.809685 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.809662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tlph8" event={"ID":"00ec9d85-c523-457e-8688-a96deb7d1ed7","Type":"ContainerStarted","Data":"d8959386b0d1ccc4e96d924d5ec6add0836c47ec67dad6688bbb316668612383"} Jul 15 19:29:01.811342 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.811317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7l44b" event={"ID":"2514b799-dd92-4c5d-9a7e-63f79cab71ef","Type":"ContainerStarted","Data":"5f9b164bd4de31ae40f94c8db0623d98489ce2d7c15112f5dddebbb025a559a2"} Jul 15 19:29:01.812936 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.812911 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" event={"ID":"eb0fb17f4c4321230ae8f9ba72a0414e","Type":"ContainerStarted","Data":"c64120c2d048d023d4dcfdde9a5b6ded536cb37568a2adb41341372906ead7fb"} Jul 15 19:29:01.814410 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.814389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" event={"ID":"5f82c026-8052-4285-899a-5df9678180a1","Type":"ContainerStarted","Data":"f89fb5c817ed91e347f856958bf208c2df4098f87b4b748869ab36361f698223"} Jul 15 19:29:01.815924 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.815901 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" event={"ID":"e9e7bed6-c4e2-4ab1-bf98-f9b398a89b9b","Type":"ContainerStarted","Data":"9102871bf2e6b205e65c134518bb93d12ad207591ef0e6881954dbb5e5c01d39"} Jul 15 19:29:01.817269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.817244 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5b2v" event={"ID":"0f36152a-ffe5-4e8d-b335-d52517232ba1","Type":"ContainerStarted","Data":"e141ee2247ee065f6aaad34e341acc40cc97cc064139d5e9253302258a620ce3"} Jul 15 19:29:01.852761 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.852723 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x5b2v" podStartSLOduration=2.537895994 podStartE2EDuration="22.852708212s" totalImagesPullingTime="20.314812218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:01.852477202 +0000 UTC m=+23.713681555" watchObservedRunningTime="2026-07-15 19:29:01.852708212 +0000 UTC m=+23.713912586" Jul 15 19:29:01.853074 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.853051 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r7mkh" podStartSLOduration=2.519914606 podStartE2EDuration="22.853045752s" totalImagesPullingTime="20.333131146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:01.840308122 +0000 UTC m=+23.701512476" watchObservedRunningTime="2026-07-15 19:29:01.853045752 +0000 UTC m=+23.714250105" Jul 15 19:29:01.880993 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.880950 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7l44b" podStartSLOduration=2.562401807 podStartE2EDuration="22.880934734s" totalImagesPullingTime="20.318532927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:01.866010867 +0000 UTC m=+23.727215243" watchObservedRunningTime="2026-07-15 19:29:01.880934734 +0000 UTC m=+23.742139087" Jul 15 19:29:01.881099 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.881050 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tlph8" podStartSLOduration=2.491625519 podStartE2EDuration="22.881043652s" totalImagesPullingTime="20.389418133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:01.880791294 +0000 UTC m=+23.741995682" watchObservedRunningTime="2026-07-15 19:29:01.881043652 +0000 UTC m=+23.742248058" Jul 15 19:29:01.892946 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.892909 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-40.ec2.internal" podStartSLOduration=21.892890806 podStartE2EDuration="21.892890806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:01.892790148 +0000 UTC m=+23.753994501" watchObservedRunningTime="2026-07-15 19:29:01.892890806 +0000 UTC m=+23.754095158" Jul 15 19:29:01.906333 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:01.906290 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mzq5g" podStartSLOduration=2.5998530779999998 podStartE2EDuration="22.906276639s" totalImagesPullingTime="20.306423561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:01.9059892 +0000 UTC m=+23.767193554" watchObservedRunningTime="2026-07-15 19:29:01.906276639 +0000 UTC m=+23.767480991" Jul 15 19:29:02.690153 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.690109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:02.690607 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:02.690289 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:29:02.822596 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.822569 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:29:02.823033 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.822992 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"7f930a0a24727a0b831ee2926288f452a125f3da342848ef45532ae2a05c0d0c"} Jul 15 19:29:02.823160 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.823094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"1011f3c3d922d55f21b54bd6c2fc12341f972467e0266906f804fc73d3d671d1"} Jul 15 19:29:02.823160 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.823157 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"e1dcb11d9e1da22be475fc6feb7abd85e5bf12b71e3a6381f2a9d76199bb22e1"} Jul 15 19:29:02.823241 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.823202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"fae4accfef9aa206d83fe990b62de9bdc239ec99e36700700f1749025d53b4a4"} Jul 15 19:29:02.824594 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.824572 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8lq9" event={"ID":"f0c390fc-7420-4fd0-9cd6-c9957b144ee6","Type":"ContainerStarted","Data":"52e7c65625f66745db227e1f139cabe57cb2eac088cc395a97e3bbfc51de9c44"} Jul 15 19:29:02.838862 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:02.838794 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q8lq9" podStartSLOduration=3.528638021 podStartE2EDuration="23.838781618s" totalImagesPullingTime="20.310143597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:02.838641627 +0000 UTC m=+24.699846000" watchObservedRunningTime="2026-07-15 19:29:02.838781618 +0000 UTC m=+24.699985972" Jul 15 19:29:03.689483 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:03.689446 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:03.689747 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:03.689456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:03.689747 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:03.689596 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:29:03.689747 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:03.689627 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:04.690029 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:04.689874 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:04.690387 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:04.690112 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:29:04.832635 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:04.832601 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:29:04.833024 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:04.832998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"357ebb5ccaf819c7670b163b2237138be20e8aee57e03c1a363e11ab6f4b130b"} Jul 15 19:29:05.352485 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:05.352450 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:29:05.353561 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:05.353542 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:29:05.689700 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:05.689662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:05.689904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:05.689702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:05.690002 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:05.689970 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:05.690082 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:05.690022 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:29:05.835868 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:05.835829 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:29:05.836322 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:05.836303 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x5b2v" Jul 15 19:29:06.467966 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.467937 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 19:29:06.613754 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.613526 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T19:29:06.467959703Z","UUID":"18bdd3b9-0fe2-4fa3-8863-3ee277da6f24","Handler":null,"Name":"","Endpoint":""} Jul 15 19:29:06.615268 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.615233 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 19:29:06.615377 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.615284 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 19:29:06.689507 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.689408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:06.689663 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:06.689544 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:29:06.841319 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.841293 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:29:06.842125 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.841737 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"65fe8e8c9923ec6f17ca47b337cd0aa78a57a8778b7fb73089e32da4cee9728e"} Jul 15 19:29:06.842125 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.841986 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:29:06.842288 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.842271 2573 scope.go:117] "RemoveContainer" containerID="e892b501da48d76e047e0aab165fd1a9a0701684e6671477f9fa105901f83e80" Jul 15 19:29:06.843968 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.843933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" event={"ID":"5f82c026-8052-4285-899a-5df9678180a1","Type":"ContainerStarted","Data":"a3ad5ac872042da827929c4d2ca2deaaf54b92601a81387f8b17db59129740b6"} Jul 15 19:29:06.847948 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.847859 2573 generic.go:356] "Generic (PLEG): container finished" podID="48b7ce9d-c732-4b21-a7ef-bd8f9963a183" containerID="50fab5143547a09cf691fea4caf8f38282e099b5d95a95cf3ee6699b1ae479b3" exitCode=0 Jul 15 19:29:06.848082 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.847944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerDied","Data":"50fab5143547a09cf691fea4caf8f38282e099b5d95a95cf3ee6699b1ae479b3"} Jul 15 19:29:06.860659 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:06.860635 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:29:07.690150 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.690116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:07.690339 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.690152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:07.690339 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:07.690240 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:29:07.690452 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:07.690383 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:07.853873 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.853841 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:29:07.854359 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.854276 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" event={"ID":"61bd4882-80fa-43e2-9dab-ad2dd5ef25d8","Type":"ContainerStarted","Data":"111caaff28f058b48e0243c41e4703f88176864effcabee96e8515defe1df58b"} Jul 15 19:29:07.854753 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.854706 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:29:07.854882 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.854793 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:29:07.870411 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.870377 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:29:07.879578 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:07.879523 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" podStartSLOduration=8.497513936 podStartE2EDuration="28.879501674s" totalImagesPullingTime="20.381987738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:07.879028362 +0000 UTC m=+29.740232729" watchObservedRunningTime="2026-07-15 19:29:07.879501674 +0000 UTC m=+29.740706029" Jul 15 19:29:08.647686 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.647542 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j5zpl"] Jul 15 19:29:08.647848 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.647839 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:08.647977 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:08.647953 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:29:08.650779 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.650754 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f9fvm"] Jul 15 19:29:08.651024 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.651006 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:08.651137 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:08.651116 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:29:08.651569 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.651549 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lwlwl"] Jul 15 19:29:08.651659 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.651646 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:08.651752 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:08.651732 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:08.859106 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.859029 2573 generic.go:356] "Generic (PLEG): container finished" podID="48b7ce9d-c732-4b21-a7ef-bd8f9963a183" containerID="a35215420b3011d1289c3baa1a54c2b0c39c574bca9be6419c25abe39846d27b" exitCode=0 Jul 15 19:29:08.859852 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.859105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerDied","Data":"a35215420b3011d1289c3baa1a54c2b0c39c574bca9be6419c25abe39846d27b"} Jul 15 19:29:08.861484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.861456 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" event={"ID":"5f82c026-8052-4285-899a-5df9678180a1","Type":"ContainerStarted","Data":"dfe08873c48e79df0b5a0260a82af3817634544aa2833a4ac5ae22029fcc2bc6"} Jul 15 19:29:08.893146 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:08.893097 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbs7h" podStartSLOduration=3.42471632 podStartE2EDuration="29.893079982s" totalImagesPullingTime="26.468363662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:08.892619088 +0000 UTC m=+30.753823452" watchObservedRunningTime="2026-07-15 19:29:08.893079982 +0000 UTC m=+30.754284334" Jul 15 19:29:10.689182 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:10.689148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:10.689182 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:10.689182 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:10.689667 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:10.689148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:10.689667 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:10.689279 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:29:10.689667 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:10.689336 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:10.689667 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:10.689406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:29:10.868462 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:10.868377 2573 generic.go:356] "Generic (PLEG): container finished" podID="48b7ce9d-c732-4b21-a7ef-bd8f9963a183" containerID="b28bc13b56960ccc5172959a18f25d69ddef5231798f9552d34086971d8033ba" exitCode=0 Jul 15 19:29:10.868462 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:10.868428 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerDied","Data":"b28bc13b56960ccc5172959a18f25d69ddef5231798f9552d34086971d8033ba"} Jul 15 19:29:12.301141 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:12.300941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:12.301611 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:12.301157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:12.301611 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.301106 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:29:12.301611 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.301270 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:29:12.301611 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.301281 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:44.301259599 +0000 UTC m=+66.162463930 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:29:12.301611 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.301309 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret podName:df75c58a-3f8a-4b70-8a59-a768bb296b53 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:44.30129509 +0000 UTC m=+66.162499421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret") pod "global-pull-secret-syncer-f9fvm" (UID: "df75c58a-3f8a-4b70-8a59-a768bb296b53") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:29:12.402018 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:12.401985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:12.402172 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.402143 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:29:12.402172 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.402162 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:29:12.402172 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.402172 2573 projected.go:196] Error preparing data for projected volume kube-api-access-772sg for pod openshift-network-diagnostics/network-check-target-lwlwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:29:12.402314 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.402226 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg podName:3a3888c7-eea2-453e-9cac-805bc3236eb4 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:44.402212221 +0000 UTC m=+66.263416553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-772sg" (UniqueName: "kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg") pod "network-check-target-lwlwl" (UID: "3a3888c7-eea2-453e-9cac-805bc3236eb4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:29:12.689401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:12.689369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:12.689583 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:12.689405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:12.689583 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.689501 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwlwl" podUID="3a3888c7-eea2-453e-9cac-805bc3236eb4" Jul 15 19:29:12.689694 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.689654 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j5zpl" podUID="7ab526b6-f5f9-4a7c-ab2b-04eb387d9520" Jul 15 19:29:12.689757 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:12.689726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:12.689843 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:12.689822 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9fvm" podUID="df75c58a-3f8a-4b70-8a59-a768bb296b53" Jul 15 19:29:13.923935 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.923904 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-40.ec2.internal" event="NodeReady" Jul 15 19:29:13.924363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.924068 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 19:29:13.970485 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.970449 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4gz28"] Jul 15 19:29:13.989307 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.989268 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-72d5n"] Jul 15 19:29:13.989460 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.989388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:13.993264 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.993031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9zc59\"" Jul 15 19:29:13.993264 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.993117 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 19:29:13.993264 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:13.993219 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 19:29:14.007422 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.007395 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-72d5n"] Jul 15 19:29:14.007539 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.007435 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4gz28"] Jul 15 19:29:14.007539 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.007498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.009547 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.009525 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jtlc5\"" Jul 15 19:29:14.009547 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.009538 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 19:29:14.009706 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.009558 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 19:29:14.009778 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.009763 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 19:29:14.113541 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.113502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-tmp-dir\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.113717 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.113556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lhn6\" (UniqueName: \"kubernetes.io/projected/bed9667d-b54b-4056-aa7b-300f7905aa67-kube-api-access-2lhn6\") pod \"ingress-canary-72d5n\" (UID: \"bed9667d-b54b-4056-aa7b-300f7905aa67\") " pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.113717 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.113654 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-config-volume\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.113717 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.113688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.113883 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.113717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bed9667d-b54b-4056-aa7b-300f7905aa67-cert\") pod \"ingress-canary-72d5n\" (UID: \"bed9667d-b54b-4056-aa7b-300f7905aa67\") " pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.113883 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.113799 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrr28\" (UniqueName: \"kubernetes.io/projected/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-kube-api-access-rrr28\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.214881 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.214839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrr28\" (UniqueName: \"kubernetes.io/projected/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-kube-api-access-rrr28\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.215087 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.214920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-tmp-dir\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.215087 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.214949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lhn6\" (UniqueName: \"kubernetes.io/projected/bed9667d-b54b-4056-aa7b-300f7905aa67-kube-api-access-2lhn6\") pod \"ingress-canary-72d5n\" (UID: \"bed9667d-b54b-4056-aa7b-300f7905aa67\") " pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.215087 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.214972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-config-volume\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.215087 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.214997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.215087 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.215022 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bed9667d-b54b-4056-aa7b-300f7905aa67-cert\") pod \"ingress-canary-72d5n\" (UID: \"bed9667d-b54b-4056-aa7b-300f7905aa67\") " pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.215397 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.215331 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-tmp-dir\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.215458 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:14.215419 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:29:14.215509 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:14.215495 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls podName:8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:14.715474897 +0000 UTC m=+36.576679236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls") pod "dns-default-4gz28" (UID: "8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6") : secret "dns-default-metrics-tls" not found Jul 15 19:29:14.215566 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.215557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-config-volume\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.219846 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.219796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bed9667d-b54b-4056-aa7b-300f7905aa67-cert\") pod \"ingress-canary-72d5n\" (UID: \"bed9667d-b54b-4056-aa7b-300f7905aa67\") " pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.224345 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.224325 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrr28\" (UniqueName: \"kubernetes.io/projected/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-kube-api-access-rrr28\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.224664 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.224645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lhn6\" (UniqueName: \"kubernetes.io/projected/bed9667d-b54b-4056-aa7b-300f7905aa67-kube-api-access-2lhn6\") pod \"ingress-canary-72d5n\" (UID: \"bed9667d-b54b-4056-aa7b-300f7905aa67\") " pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.317777 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.317740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-72d5n" Jul 15 19:29:14.689539 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.689504 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:14.689775 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.689508 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:14.689775 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.689509 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:14.692234 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.692214 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:29:14.692339 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.692274 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:29:14.692385 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.692214 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:29:14.692985 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.692965 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5v82\"" Jul 15 19:29:14.693271 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.693245 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:29:14.693271 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.693261 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jvdh\"" Jul 15 19:29:14.719623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:14.719599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:14.719781 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:14.719764 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:29:14.719868 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:14.719856 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls podName:8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:15.719838387 +0000 UTC m=+37.581042721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls") pod "dns-default-4gz28" (UID: "8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6") : secret "dns-default-metrics-tls" not found Jul 15 19:29:15.131321 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:15.131290 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-72d5n"] Jul 15 19:29:15.180431 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:15.180401 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed9667d_b54b_4056_aa7b_300f7905aa67.slice/crio-121af92e7ec7ae2282acab20abc6dfa96ed35f6d78a0354e645d7ced4c4323d8 WatchSource:0}: Error finding container 121af92e7ec7ae2282acab20abc6dfa96ed35f6d78a0354e645d7ced4c4323d8: Status 404 returned error can't find the container with id 121af92e7ec7ae2282acab20abc6dfa96ed35f6d78a0354e645d7ced4c4323d8 Jul 15 19:29:15.728874 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:15.728626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:15.729034 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:15.728837 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:29:15.729034 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:15.729014 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls podName:8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:17.728997639 +0000 UTC m=+39.590201970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls") pod "dns-default-4gz28" (UID: "8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6") : secret "dns-default-metrics-tls" not found Jul 15 19:29:15.883234 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:15.883191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-72d5n" event={"ID":"bed9667d-b54b-4056-aa7b-300f7905aa67","Type":"ContainerStarted","Data":"121af92e7ec7ae2282acab20abc6dfa96ed35f6d78a0354e645d7ced4c4323d8"} Jul 15 19:29:15.886422 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:15.886392 2573 generic.go:356] "Generic (PLEG): container finished" podID="48b7ce9d-c732-4b21-a7ef-bd8f9963a183" containerID="d52370568870335ecb9c67f91e6285bd7d807446946a58475df68e97c6f9431d" exitCode=0 Jul 15 19:29:15.886579 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:15.886442 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerDied","Data":"d52370568870335ecb9c67f91e6285bd7d807446946a58475df68e97c6f9431d"} Jul 15 19:29:16.892487 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:16.892448 2573 generic.go:356] "Generic (PLEG): container finished" podID="48b7ce9d-c732-4b21-a7ef-bd8f9963a183" containerID="2c842f3ff80023adafa7021ac0e472b6bedeb2f771e8f92ef7e22740cca806ad" exitCode=0 Jul 15 19:29:16.892979 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:16.892516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerDied","Data":"2c842f3ff80023adafa7021ac0e472b6bedeb2f771e8f92ef7e22740cca806ad"} Jul 15 19:29:17.746070 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:17.745984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:17.746214 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:17.746141 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:29:17.746214 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:17.746201 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls podName:8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:21.74618492 +0000 UTC m=+43.607389252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls") pod "dns-default-4gz28" (UID: "8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6") : secret "dns-default-metrics-tls" not found Jul 15 19:29:17.896145 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:17.896109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-72d5n" event={"ID":"bed9667d-b54b-4056-aa7b-300f7905aa67","Type":"ContainerStarted","Data":"2eb2cd5fba7f49af3df6a364a8f25777bfe5457b8b961e067d0b276a96b17c65"} Jul 15 19:29:17.899721 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:17.899690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" event={"ID":"48b7ce9d-c732-4b21-a7ef-bd8f9963a183","Type":"ContainerStarted","Data":"87f424d1c3e2511541f24655f3ea6298ec2296295c2956b853d16068922b68e1"} Jul 15 19:29:17.910112 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:17.910069 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-72d5n" podStartSLOduration=2.595194807 podStartE2EDuration="4.910053611s" totalImagesPullingTime="2.314858804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:17.908940578 +0000 UTC m=+39.770144953" watchObservedRunningTime="2026-07-15 19:29:17.910053611 +0000 UTC m=+39.771257963" Jul 15 19:29:17.929064 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:17.929019 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gfr4h" podStartSLOduration=10.557101121 podStartE2EDuration="38.929004738s" totalImagesPullingTime="28.371903617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:17.928854326 +0000 UTC m=+39.790058708" watchObservedRunningTime="2026-07-15 19:29:17.929004738 +0000 UTC m=+39.790209085" Jul 15 19:29:21.775542 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:21.775496 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:21.776058 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:21.775610 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:29:21.776058 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:21.775662 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls podName:8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:29.775648234 +0000 UTC m=+51.636852565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls") pod "dns-default-4gz28" (UID: "8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6") : secret "dns-default-metrics-tls" not found Jul 15 19:29:29.834007 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:29.833965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:29.834495 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:29.834096 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:29:29.834495 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:29.834157 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls podName:8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:45.834144279 +0000 UTC m=+67.695348611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls") pod "dns-default-4gz28" (UID: "8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6") : secret "dns-default-metrics-tls" not found Jul 15 19:29:38.427205 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.427168 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv"] Jul 15 19:29:38.429563 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.429544 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" Jul 15 19:29:38.431677 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.431657 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:38.432296 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.432275 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2ck4d\"" Jul 15 19:29:38.433118 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.433078 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 15 19:29:38.435719 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.435697 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94"] Jul 15 19:29:38.437505 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.437488 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-mt85z"] Jul 15 19:29:38.437624 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.437608 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.439008 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.438988 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl"] Jul 15 19:29:38.439156 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.439079 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.440125 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.440104 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 15 19:29:38.440571 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.440550 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bx5fv\"" Jul 15 19:29:38.440740 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.440725 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 15 19:29:38.440937 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.440919 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv"] Jul 15 19:29:38.441110 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.441091 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 15 19:29:38.441201 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.441110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" Jul 15 19:29:38.441201 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.441133 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 15 19:29:38.441334 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.441232 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:38.441334 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.441330 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:38.441857 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.441838 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 15 19:29:38.442092 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.442075 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gc5z9\"" Jul 15 19:29:38.442233 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.442216 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 15 19:29:38.443554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.443510 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tczn5\"" Jul 15 19:29:38.454389 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.454363 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 15 19:29:38.457156 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.457129 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl"] Jul 15 19:29:38.458295 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.458278 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94"] Jul 15 19:29:38.458594 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.458572 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mt85z"] Jul 15 19:29:38.492202 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c60bcad4-3d75-45aa-a076-1c84891736f5-config\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.492382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdf2\" (UniqueName: \"kubernetes.io/projected/627fbf81-2e23-47ca-9a57-f577caf7e3f5-kube-api-access-dbdf2\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.492382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzjh\" (UniqueName: \"kubernetes.io/projected/2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b-kube-api-access-7tzjh\") pod \"volume-data-source-validator-78495597f6-8rzjv\" (UID: \"2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" Jul 15 19:29:38.492382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c60bcad4-3d75-45aa-a076-1c84891736f5-trusted-ca\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.492382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvf8\" (UniqueName: \"kubernetes.io/projected/c60bcad4-3d75-45aa-a076-1c84891736f5-kube-api-access-lgvf8\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.492382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/627fbf81-2e23-47ca-9a57-f577caf7e3f5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.492382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwq9\" (UniqueName: \"kubernetes.io/projected/a3d68494-6fc1-491e-9c23-fb2f7001df0b-kube-api-access-hcwq9\") pod \"network-check-source-869dcfcdcd-r2cdl\" (UID: \"a3d68494-6fc1-491e-9c23-fb2f7001df0b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" Jul 15 19:29:38.492621 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/627fbf81-2e23-47ca-9a57-f577caf7e3f5-config\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.492621 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.492486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c60bcad4-3d75-45aa-a076-1c84891736f5-serving-cert\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.534536 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.534505 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-95x5f"] Jul 15 19:29:38.536710 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.536689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:38.538994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.538971 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 15 19:29:38.538994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.538988 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hwqkm\"" Jul 15 19:29:38.539261 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.539245 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 15 19:29:38.552921 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.552895 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-95x5f"] Jul 15 19:29:38.592686 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c60bcad4-3d75-45aa-a076-1c84891736f5-config\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.592686 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdf2\" (UniqueName: \"kubernetes.io/projected/627fbf81-2e23-47ca-9a57-f577caf7e3f5-kube-api-access-dbdf2\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.592963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4df817f-0d0b-4c52-8f92-d12a27c50869-nginx-conf\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:38.592963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzjh\" (UniqueName: \"kubernetes.io/projected/2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b-kube-api-access-7tzjh\") pod \"volume-data-source-validator-78495597f6-8rzjv\" (UID: \"2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" Jul 15 19:29:38.592963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c60bcad4-3d75-45aa-a076-1c84891736f5-trusted-ca\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.592963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvf8\" (UniqueName: \"kubernetes.io/projected/c60bcad4-3d75-45aa-a076-1c84891736f5-kube-api-access-lgvf8\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.592963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/627fbf81-2e23-47ca-9a57-f577caf7e3f5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.593208 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.592980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:38.593208 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.593020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwq9\" (UniqueName: \"kubernetes.io/projected/a3d68494-6fc1-491e-9c23-fb2f7001df0b-kube-api-access-hcwq9\") pod \"network-check-source-869dcfcdcd-r2cdl\" (UID: \"a3d68494-6fc1-491e-9c23-fb2f7001df0b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" Jul 15 19:29:38.593208 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.593063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/627fbf81-2e23-47ca-9a57-f577caf7e3f5-config\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.593208 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.593095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c60bcad4-3d75-45aa-a076-1c84891736f5-serving-cert\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.595269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.595253 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 15 19:29:38.595333 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.595258 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 15 19:29:38.595333 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.595291 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 15 19:29:38.595333 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.595301 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 15 19:29:38.600492 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.600471 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 15 19:29:38.603291 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.603271 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 15 19:29:38.603961 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.603709 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/627fbf81-2e23-47ca-9a57-f577caf7e3f5-config\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.603961 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.603749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c60bcad4-3d75-45aa-a076-1c84891736f5-config\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.603961 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.603786 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 15 19:29:38.603961 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.603863 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 15 19:29:38.603961 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.603902 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:29:38.603961 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.603900 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c60bcad4-3d75-45aa-a076-1c84891736f5-trusted-ca\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.606789 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.606767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/627fbf81-2e23-47ca-9a57-f577caf7e3f5-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.606912 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.606893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c60bcad4-3d75-45aa-a076-1c84891736f5-serving-cert\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.615112 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.615090 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:38.615112 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.615096 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:29:38.615112 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.615107 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:38.615391 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.615379 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:38.624625 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.624603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzjh\" (UniqueName: \"kubernetes.io/projected/2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b-kube-api-access-7tzjh\") pod \"volume-data-source-validator-78495597f6-8rzjv\" (UID: \"2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" Jul 15 19:29:38.624723 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.624651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvf8\" (UniqueName: \"kubernetes.io/projected/c60bcad4-3d75-45aa-a076-1c84891736f5-kube-api-access-lgvf8\") pod \"console-operator-7866497494-mt85z\" (UID: \"c60bcad4-3d75-45aa-a076-1c84891736f5\") " pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.624723 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.624668 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdf2\" (UniqueName: \"kubernetes.io/projected/627fbf81-2e23-47ca-9a57-f577caf7e3f5-kube-api-access-dbdf2\") pod \"service-ca-operator-7695fc6c8c-qlp94\" (UID: \"627fbf81-2e23-47ca-9a57-f577caf7e3f5\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.624816 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.624746 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwq9\" (UniqueName: \"kubernetes.io/projected/a3d68494-6fc1-491e-9c23-fb2f7001df0b-kube-api-access-hcwq9\") pod \"network-check-source-869dcfcdcd-r2cdl\" (UID: \"a3d68494-6fc1-491e-9c23-fb2f7001df0b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" Jul 15 19:29:38.693558 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.693462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4df817f-0d0b-4c52-8f92-d12a27c50869-nginx-conf\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:38.693558 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.693544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:38.693788 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:38.693674 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:29:38.693788 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:38.693743 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert podName:a4df817f-0d0b-4c52-8f92-d12a27c50869 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:39.193723589 +0000 UTC m=+61.054927920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-95x5f" (UID: "a4df817f-0d0b-4c52-8f92-d12a27c50869") : secret "networking-console-plugin-cert" not found Jul 15 19:29:38.694277 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.694244 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4df817f-0d0b-4c52-8f92-d12a27c50869-nginx-conf\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:38.741511 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.741480 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2ck4d\"" Jul 15 19:29:38.750074 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.750049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" Jul 15 19:29:38.752576 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.752553 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bx5fv\"" Jul 15 19:29:38.760536 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.760520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" Jul 15 19:29:38.763312 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.763284 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gc5z9\"" Jul 15 19:29:38.768043 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.768024 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tczn5\"" Jul 15 19:29:38.771672 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.771643 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:38.776639 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.776600 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" Jul 15 19:29:38.928859 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.928832 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv"] Jul 15 19:29:38.933082 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:38.932499 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed6d1d4_eb46_4b34_b127_f5737e8a8c1b.slice/crio-e532684126b8adb9be9cd4a4d356919919156243a1327f68fc7e5171d93701b8 WatchSource:0}: Error finding container e532684126b8adb9be9cd4a4d356919919156243a1327f68fc7e5171d93701b8: Status 404 returned error can't find the container with id e532684126b8adb9be9cd4a4d356919919156243a1327f68fc7e5171d93701b8 Jul 15 19:29:38.938313 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.938236 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94"] Jul 15 19:29:38.943142 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:38.943114 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627fbf81_2e23_47ca_9a57_f577caf7e3f5.slice/crio-c6f73519485931508cb4432dd9f935d2106cab18f62fbb348982c0488fa502c5 WatchSource:0}: Error finding container c6f73519485931508cb4432dd9f935d2106cab18f62fbb348982c0488fa502c5: Status 404 returned error can't find the container with id c6f73519485931508cb4432dd9f935d2106cab18f62fbb348982c0488fa502c5 Jul 15 19:29:38.949994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.949962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" event={"ID":"627fbf81-2e23-47ca-9a57-f577caf7e3f5","Type":"ContainerStarted","Data":"c6f73519485931508cb4432dd9f935d2106cab18f62fbb348982c0488fa502c5"} Jul 15 19:29:38.951787 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.951762 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" event={"ID":"2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b","Type":"ContainerStarted","Data":"e532684126b8adb9be9cd4a4d356919919156243a1327f68fc7e5171d93701b8"} Jul 15 19:29:38.956907 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.956885 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl"] Jul 15 19:29:38.959073 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:38.959044 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d68494_6fc1_491e_9c23_fb2f7001df0b.slice/crio-82eb44e8b9ee082d4fa92223a3109ccab5cf0ee1cec0df7678d07e9e66e3f2e8 WatchSource:0}: Error finding container 82eb44e8b9ee082d4fa92223a3109ccab5cf0ee1cec0df7678d07e9e66e3f2e8: Status 404 returned error can't find the container with id 82eb44e8b9ee082d4fa92223a3109ccab5cf0ee1cec0df7678d07e9e66e3f2e8 Jul 15 19:29:38.971839 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:38.971790 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mt85z"] Jul 15 19:29:38.974759 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:38.974735 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60bcad4_3d75_45aa_a076_1c84891736f5.slice/crio-30146d92d4172f822e7cdb313eaf9abd60db7be83a08f546997bf6d6df5a37bb WatchSource:0}: Error finding container 30146d92d4172f822e7cdb313eaf9abd60db7be83a08f546997bf6d6df5a37bb: Status 404 returned error can't find the container with id 30146d92d4172f822e7cdb313eaf9abd60db7be83a08f546997bf6d6df5a37bb Jul 15 19:29:39.201752 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:39.201660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:39.201923 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:39.201784 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:29:39.201923 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:39.201884 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert podName:a4df817f-0d0b-4c52-8f92-d12a27c50869 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:40.201865228 +0000 UTC m=+62.063069563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-95x5f" (UID: "a4df817f-0d0b-4c52-8f92-d12a27c50869") : secret "networking-console-plugin-cert" not found Jul 15 19:29:39.879480 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:39.879446 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qp4s" Jul 15 19:29:39.956600 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:39.956516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" event={"ID":"a3d68494-6fc1-491e-9c23-fb2f7001df0b","Type":"ContainerStarted","Data":"82eb44e8b9ee082d4fa92223a3109ccab5cf0ee1cec0df7678d07e9e66e3f2e8"} Jul 15 19:29:39.961436 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:39.961404 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mt85z" event={"ID":"c60bcad4-3d75-45aa-a076-1c84891736f5","Type":"ContainerStarted","Data":"30146d92d4172f822e7cdb313eaf9abd60db7be83a08f546997bf6d6df5a37bb"} Jul 15 19:29:40.209702 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:40.209661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:40.209906 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:40.209868 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:29:40.209970 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:40.209937 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert podName:a4df817f-0d0b-4c52-8f92-d12a27c50869 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:42.209916417 +0000 UTC m=+64.071120758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-95x5f" (UID: "a4df817f-0d0b-4c52-8f92-d12a27c50869") : secret "networking-console-plugin-cert" not found Jul 15 19:29:40.966304 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:40.966263 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" event={"ID":"2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b","Type":"ContainerStarted","Data":"343dfd768bbff4249f8ffaec4aa347757a4cda6d0a4751266ca65b8c7de0fa26"} Jul 15 19:29:42.227372 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.227328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:42.227775 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:42.227475 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:29:42.227775 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:42.227544 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert podName:a4df817f-0d0b-4c52-8f92-d12a27c50869 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:46.227528349 +0000 UTC m=+68.088732680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-95x5f" (UID: "a4df817f-0d0b-4c52-8f92-d12a27c50869") : secret "networking-console-plugin-cert" not found Jul 15 19:29:42.832161 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.832102 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-8rzjv" podStartSLOduration=3.278670422 podStartE2EDuration="4.832082405s" totalImagesPullingTime="1.553411983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:40.979251791 +0000 UTC m=+62.840456155" watchObservedRunningTime="2026-07-15 19:29:42.832082405 +0000 UTC m=+64.693286761" Jul 15 19:29:42.833018 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.832990 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk"] Jul 15 19:29:42.835760 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.835743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" Jul 15 19:29:42.838428 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.838003 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 15 19:29:42.838428 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.838254 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 15 19:29:42.838715 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.838696 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tq2z8\"" Jul 15 19:29:42.845344 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.845323 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk"] Jul 15 19:29:42.934236 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:42.934197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzvwf\" (UniqueName: \"kubernetes.io/projected/07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1-kube-api-access-gzvwf\") pod \"migrator-7f59b54f6c-xgkxk\" (UID: \"07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" Jul 15 19:29:43.034899 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.034859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzvwf\" (UniqueName: \"kubernetes.io/projected/07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1-kube-api-access-gzvwf\") pod \"migrator-7f59b54f6c-xgkxk\" (UID: \"07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" Jul 15 19:29:43.043733 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.043705 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzvwf\" (UniqueName: \"kubernetes.io/projected/07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1-kube-api-access-gzvwf\") pod \"migrator-7f59b54f6c-xgkxk\" (UID: \"07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" Jul 15 19:29:43.147164 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.147069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" Jul 15 19:29:43.351556 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.351523 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk"] Jul 15 19:29:43.355253 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:43.355218 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a2f8ca_1c14_465d_a6d7_2a1ad7b228f1.slice/crio-6a2522d4d61ac334e99a6db64c4e62e66a374ce522624ad2c50d704dbd4a7ddb WatchSource:0}: Error finding container 6a2522d4d61ac334e99a6db64c4e62e66a374ce522624ad2c50d704dbd4a7ddb: Status 404 returned error can't find the container with id 6a2522d4d61ac334e99a6db64c4e62e66a374ce522624ad2c50d704dbd4a7ddb Jul 15 19:29:43.359722 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.359702 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5w487"] Jul 15 19:29:43.362510 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.362183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.365265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.364369 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wwfmj\"" Jul 15 19:29:43.365265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.364401 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 19:29:43.365265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.364514 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 19:29:43.365265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.364598 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 19:29:43.365265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.365103 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 19:29:43.371974 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.371658 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5w487"] Jul 15 19:29:43.439045 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.439016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l48nk\" (UniqueName: \"kubernetes.io/projected/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-api-access-l48nk\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.439196 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.439066 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.439196 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.439165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-crio-socket\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.439269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.439203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.439308 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.439280 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-data-volume\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540274 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.540235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-data-volume\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540434 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.540328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48nk\" (UniqueName: \"kubernetes.io/projected/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-api-access-l48nk\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540434 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.540379 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540540 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.540448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-crio-socket\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540540 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.540474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540631 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.540549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-crio-socket\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540631 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:43.540579 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:29:43.540777 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.540630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-data-volume\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.540777 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:43.540661 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:44.04063987 +0000 UTC m=+65.901844212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : secret "insights-runtime-extractor-tls" not found Jul 15 19:29:43.540777 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:43.540579 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:29:43.540777 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:43.540731 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:44.040712697 +0000 UTC m=+65.901917031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : configmap "kube-rbac-proxy" not found Jul 15 19:29:43.548012 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.547988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48nk\" (UniqueName: \"kubernetes.io/projected/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-api-access-l48nk\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:43.981023 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.980979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" event={"ID":"627fbf81-2e23-47ca-9a57-f577caf7e3f5","Type":"ContainerStarted","Data":"12c0db6cf3dd71fa481af7b030088238514f9906510b68ba8fea6f18dff6bb4c"} Jul 15 19:29:43.982593 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.982562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" event={"ID":"a3d68494-6fc1-491e-9c23-fb2f7001df0b","Type":"ContainerStarted","Data":"5c8a7473be52781b13c3b864957a8bd026fd31f4fdbed7eb6a768927f9e39ce7"} Jul 15 19:29:43.984444 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.984420 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/0.log" Jul 15 19:29:43.984558 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.984463 2573 generic.go:356] "Generic (PLEG): container finished" podID="c60bcad4-3d75-45aa-a076-1c84891736f5" containerID="fafdd1f94fde9086f5a25693d172fea203068eaed617f4357cd2c8e2a8ac0f37" exitCode=255 Jul 15 19:29:43.984618 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.984552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mt85z" event={"ID":"c60bcad4-3d75-45aa-a076-1c84891736f5","Type":"ContainerDied","Data":"fafdd1f94fde9086f5a25693d172fea203068eaed617f4357cd2c8e2a8ac0f37"} Jul 15 19:29:43.984720 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.984702 2573 scope.go:117] "RemoveContainer" containerID="fafdd1f94fde9086f5a25693d172fea203068eaed617f4357cd2c8e2a8ac0f37" Jul 15 19:29:43.985938 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.985919 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" event={"ID":"07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1","Type":"ContainerStarted","Data":"6a2522d4d61ac334e99a6db64c4e62e66a374ce522624ad2c50d704dbd4a7ddb"} Jul 15 19:29:43.997143 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:43.997096 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" podStartSLOduration=1.734864153 podStartE2EDuration="5.997080109s" totalImagesPullingTime="4.262215956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:43.995886929 +0000 UTC m=+65.857091301" watchObservedRunningTime="2026-07-15 19:29:43.997080109 +0000 UTC m=+65.858284463" Jul 15 19:29:44.009314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.009246 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-r2cdl" podStartSLOduration=1.752088268 podStartE2EDuration="6.009227997s" totalImagesPullingTime="4.257139729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:44.008396413 +0000 UTC m=+65.869600768" watchObservedRunningTime="2026-07-15 19:29:44.009227997 +0000 UTC m=+65.870432343" Jul 15 19:29:44.045456 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.045386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:44.045532 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.045509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:44.045757 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:44.045717 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:29:44.045876 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:44.045794 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:45.045775848 +0000 UTC m=+66.906980178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : secret "insights-runtime-extractor-tls" not found Jul 15 19:29:44.046862 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:44.046830 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:29:44.047153 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:44.047136 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:45.047114354 +0000 UTC m=+66.908318688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : configmap "kube-rbac-proxy" not found Jul 15 19:29:44.350133 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.350023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:44.350336 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.350168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:29:44.352231 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.352196 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:29:44.352608 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.352263 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:29:44.361148 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:44.361118 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 19:29:44.361297 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:44.361207 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs podName:7ab526b6-f5f9-4a7c-ab2b-04eb387d9520 nodeName:}" failed. No retries permitted until 2026-07-15 19:30:48.361183727 +0000 UTC m=+130.222388058 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs") pod "network-metrics-daemon-j5zpl" (UID: "7ab526b6-f5f9-4a7c-ab2b-04eb387d9520") : secret "metrics-daemon-secret" not found Jul 15 19:29:44.363917 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.363890 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/df75c58a-3f8a-4b70-8a59-a768bb296b53-original-pull-secret\") pod \"global-pull-secret-syncer-f9fvm\" (UID: \"df75c58a-3f8a-4b70-8a59-a768bb296b53\") " pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:44.408936 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.408904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9fvm" Jul 15 19:29:44.450975 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.450940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:44.454152 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.454126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772sg\" (UniqueName: \"kubernetes.io/projected/3a3888c7-eea2-453e-9cac-805bc3236eb4-kube-api-access-772sg\") pod \"network-check-target-lwlwl\" (UID: \"3a3888c7-eea2-453e-9cac-805bc3236eb4\") " pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:44.554955 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.554927 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f9fvm"] Jul 15 19:29:44.716507 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.716480 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jvdh\"" Jul 15 19:29:44.717038 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:44.717012 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf75c58a_3f8a_4b70_8a59_a768bb296b53.slice/crio-cd94cd601371a95f39d17b409e82436cc73bbd8cd0aa90839ca0d3ec476c5c27 WatchSource:0}: Error finding container cd94cd601371a95f39d17b409e82436cc73bbd8cd0aa90839ca0d3ec476c5c27: Status 404 returned error can't find the container with id cd94cd601371a95f39d17b409e82436cc73bbd8cd0aa90839ca0d3ec476c5c27 Jul 15 19:29:44.725184 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.725164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:44.867028 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.867003 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lwlwl"] Jul 15 19:29:44.870275 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:29:44.870233 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3888c7_eea2_453e_9cac_805bc3236eb4.slice/crio-3b7aa385f95af5eb01a6c2afef6d511bf0a3299b6621353a6faaaaf771ba53d8 WatchSource:0}: Error finding container 3b7aa385f95af5eb01a6c2afef6d511bf0a3299b6621353a6faaaaf771ba53d8: Status 404 returned error can't find the container with id 3b7aa385f95af5eb01a6c2afef6d511bf0a3299b6621353a6faaaaf771ba53d8 Jul 15 19:29:44.990776 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.990705 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:29:44.991159 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.991145 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/0.log" Jul 15 19:29:44.991211 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.991180 2573 generic.go:356] "Generic (PLEG): container finished" podID="c60bcad4-3d75-45aa-a076-1c84891736f5" containerID="f2cc57c568414f92a1890603d506711eb2c4510e85c234aea318ec35f74819e1" exitCode=255 Jul 15 19:29:44.991298 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.991261 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mt85z" event={"ID":"c60bcad4-3d75-45aa-a076-1c84891736f5","Type":"ContainerDied","Data":"f2cc57c568414f92a1890603d506711eb2c4510e85c234aea318ec35f74819e1"} Jul 15 19:29:44.991423 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.991334 2573 scope.go:117] "RemoveContainer" containerID="fafdd1f94fde9086f5a25693d172fea203068eaed617f4357cd2c8e2a8ac0f37" Jul 15 19:29:44.991540 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.991525 2573 scope.go:117] "RemoveContainer" containerID="f2cc57c568414f92a1890603d506711eb2c4510e85c234aea318ec35f74819e1" Jul 15 19:29:44.991736 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:44.991715 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mt85z_openshift-console-operator(c60bcad4-3d75-45aa-a076-1c84891736f5)\"" pod="openshift-console-operator/console-operator-7866497494-mt85z" podUID="c60bcad4-3d75-45aa-a076-1c84891736f5" Jul 15 19:29:44.993107 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.993089 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" event={"ID":"07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1","Type":"ContainerStarted","Data":"998a300ad20dd591d058e1c458f2e938ea2ad7115432f7b9334e96d43674e8ed"} Jul 15 19:29:44.993180 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.993143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" event={"ID":"07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1","Type":"ContainerStarted","Data":"44798c4fb8ec06ca95ce7debd7146fc751ee0f2defd3c21434a50d6af9ed4889"} Jul 15 19:29:44.994639 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.994610 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lwlwl" event={"ID":"3a3888c7-eea2-453e-9cac-805bc3236eb4","Type":"ContainerStarted","Data":"ce5aab6b85ca872a903777e66adbc777e62d8bb3554ff6509f1c403d5429bb04"} Jul 15 19:29:44.994787 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.994644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lwlwl" event={"ID":"3a3888c7-eea2-453e-9cac-805bc3236eb4","Type":"ContainerStarted","Data":"3b7aa385f95af5eb01a6c2afef6d511bf0a3299b6621353a6faaaaf771ba53d8"} Jul 15 19:29:44.994787 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.994664 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:29:44.995732 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:44.995699 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f9fvm" event={"ID":"df75c58a-3f8a-4b70-8a59-a768bb296b53","Type":"ContainerStarted","Data":"cd94cd601371a95f39d17b409e82436cc73bbd8cd0aa90839ca0d3ec476c5c27"} Jul 15 19:29:45.018589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:45.018544 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xgkxk" podStartSLOduration=1.635411143 podStartE2EDuration="3.018530616s" totalImagesPullingTime="1.383119473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:45.018010032 +0000 UTC m=+66.879214397" watchObservedRunningTime="2026-07-15 19:29:45.018530616 +0000 UTC m=+66.879734969" Jul 15 19:29:45.033441 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:45.033396 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lwlwl" podStartSLOduration=66.033383575 podStartE2EDuration="1m6.033383575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:45.03278139 +0000 UTC m=+66.893985776" watchObservedRunningTime="2026-07-15 19:29:45.033383575 +0000 UTC m=+66.894587963" Jul 15 19:29:45.055150 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:45.055123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:45.055303 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:45.055200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:45.055303 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:45.055276 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:29:45.055418 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:45.055341 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:47.05532099 +0000 UTC m=+68.916525340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : configmap "kube-rbac-proxy" not found Jul 15 19:29:45.055418 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:45.055405 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:29:45.055541 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:45.055448 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:47.055436036 +0000 UTC m=+68.916640371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : secret "insights-runtime-extractor-tls" not found Jul 15 19:29:45.868556 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:45.868511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:29:45.869047 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:45.868726 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:29:45.869047 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:45.868845 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls podName:8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6 nodeName:}" failed. No retries permitted until 2026-07-15 19:30:17.86879333 +0000 UTC m=+99.729997666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls") pod "dns-default-4gz28" (UID: "8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6") : secret "dns-default-metrics-tls" not found Jul 15 19:29:45.953143 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:45.953113 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7l44b_2514b799-dd92-4c5d-9a7e-63f79cab71ef/dns-node-resolver/0.log" Jul 15 19:29:46.002492 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:46.002456 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:29:46.003050 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:46.003026 2573 scope.go:117] "RemoveContainer" containerID="f2cc57c568414f92a1890603d506711eb2c4510e85c234aea318ec35f74819e1" Jul 15 19:29:46.003187 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:46.003173 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mt85z_openshift-console-operator(c60bcad4-3d75-45aa-a076-1c84891736f5)\"" pod="openshift-console-operator/console-operator-7866497494-mt85z" podUID="c60bcad4-3d75-45aa-a076-1c84891736f5" Jul 15 19:29:46.272488 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:46.272450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:46.272670 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:46.272602 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:29:46.272717 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:46.272672 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert podName:a4df817f-0d0b-4c52-8f92-d12a27c50869 nodeName:}" failed. No retries permitted until 2026-07-15 19:29:54.272653376 +0000 UTC m=+76.133857728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-95x5f" (UID: "a4df817f-0d0b-4c52-8f92-d12a27c50869") : secret "networking-console-plugin-cert" not found Jul 15 19:29:47.080008 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:47.079974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:47.080425 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:47.080055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:47.080425 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:47.080176 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:29:47.080425 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:47.080238 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:51.080218349 +0000 UTC m=+72.941422683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : secret "insights-runtime-extractor-tls" not found Jul 15 19:29:47.080582 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:47.080569 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:29:47.080637 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:47.080614 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:51.080599731 +0000 UTC m=+72.941804068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : configmap "kube-rbac-proxy" not found Jul 15 19:29:47.352569 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:47.352492 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzq5g_cc2dcea3-f197-4b33-b77f-00716c3c698c/node-ca/0.log" Jul 15 19:29:47.954070 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:47.954042 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-72d5n_bed9667d-b54b-4056-aa7b-300f7905aa67/serve-healthcheck-canary/0.log" Jul 15 19:29:48.354062 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:48.353987 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xgkxk_07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1/migrator/0.log" Jul 15 19:29:48.553579 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:48.553539 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xgkxk_07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1/graceful-termination/0.log" Jul 15 19:29:48.772252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:48.772215 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:48.772458 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:48.772291 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:29:48.772696 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:48.772682 2573 scope.go:117] "RemoveContainer" containerID="f2cc57c568414f92a1890603d506711eb2c4510e85c234aea318ec35f74819e1" Jul 15 19:29:48.772912 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:48.772882 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mt85z_openshift-console-operator(c60bcad4-3d75-45aa-a076-1c84891736f5)\"" pod="openshift-console-operator/console-operator-7866497494-mt85z" podUID="c60bcad4-3d75-45aa-a076-1c84891736f5" Jul 15 19:29:50.017281 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:50.017245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f9fvm" event={"ID":"df75c58a-3f8a-4b70-8a59-a768bb296b53","Type":"ContainerStarted","Data":"c7ff436cc96ee9e73688e0674ed1c9e1bbfdf5a50bbc7695bae0146e06c551d1"} Jul 15 19:29:50.031185 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:50.031138 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f9fvm" podStartSLOduration=66.545968863 podStartE2EDuration="1m11.031123269s" totalImagesPullingTime="4.485154406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:29:50.030684982 +0000 UTC m=+71.891889335" watchObservedRunningTime="2026-07-15 19:29:50.031123269 +0000 UTC m=+71.892327622" Jul 15 19:29:51.113714 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:51.113662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:51.114316 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:51.113755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:51.114316 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:51.113844 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:29:51.114316 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:51.113875 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:29:51.114316 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:51.113914 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:59.113895879 +0000 UTC m=+80.975100217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : secret "insights-runtime-extractor-tls" not found Jul 15 19:29:51.114316 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:51.113933 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:29:59.113924717 +0000 UTC m=+80.975129048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : configmap "kube-rbac-proxy" not found Jul 15 19:29:54.338749 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:54.338708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:29:54.339192 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:54.338843 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:29:54.339192 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:54.338904 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert podName:a4df817f-0d0b-4c52-8f92-d12a27c50869 nodeName:}" failed. No retries permitted until 2026-07-15 19:30:10.338886615 +0000 UTC m=+92.200090948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-95x5f" (UID: "a4df817f-0d0b-4c52-8f92-d12a27c50869") : secret "networking-console-plugin-cert" not found Jul 15 19:29:59.176606 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:59.176557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:59.177139 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:59.176637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:29:59.177139 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:59.176709 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:29:59.177139 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:29:59.176776 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls podName:ca6fa370-fa41-45c9-8b77-d0eeb61379ec nodeName:}" failed. No retries permitted until 2026-07-15 19:30:15.176760444 +0000 UTC m=+97.037964778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w487" (UID: "ca6fa370-fa41-45c9-8b77-d0eeb61379ec") : secret "insights-runtime-extractor-tls" not found Jul 15 19:29:59.177315 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:29:59.177252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:30:00.689898 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:00.689742 2573 scope.go:117] "RemoveContainer" containerID="f2cc57c568414f92a1890603d506711eb2c4510e85c234aea318ec35f74819e1" Jul 15 19:30:01.053468 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:01.053376 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:30:01.053632 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:01.053498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mt85z" event={"ID":"c60bcad4-3d75-45aa-a076-1c84891736f5","Type":"ContainerStarted","Data":"d36702f722fda8f5d76efd5acd00d31c3b8e46d917e1af871958d701f2f01fa0"} Jul 15 19:30:01.054170 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:01.054139 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:30:01.064261 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:01.064234 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-mt85z" Jul 15 19:30:01.070824 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:01.070751 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-mt85z" podStartSLOduration=18.842986413 podStartE2EDuration="23.07073685s" totalImagesPullingTime="4.227750437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:01.06986718 +0000 UTC m=+82.931071557" watchObservedRunningTime="2026-07-15 19:30:01.07073685 +0000 UTC m=+82.931941223" Jul 15 19:30:08.050541 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.050506 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln"] Jul 15 19:30:08.053684 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.053662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.056019 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.055997 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 19:30:08.056264 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.056243 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 19:30:08.057032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.057013 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 19:30:08.057143 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.057094 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 19:30:08.057143 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.057111 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 19:30:08.057143 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.057132 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 19:30:08.057422 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.057407 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 19:30:08.066488 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.066467 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln"] Jul 15 19:30:08.147635 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.147602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-hub\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.147854 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.147688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3187b710-1361-401b-a6e8-f652a1285796-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.147854 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.147719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.147854 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.147771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-ca\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.147854 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.147821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.148026 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.147863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kk8j\" (UniqueName: \"kubernetes.io/projected/3187b710-1361-401b-a6e8-f652a1285796-kube-api-access-9kk8j\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.248688 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.248652 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3187b710-1361-401b-a6e8-f652a1285796-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.248688 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.248686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.248988 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.248722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-ca\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.248988 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.248744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.248988 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.248766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kk8j\" (UniqueName: \"kubernetes.io/projected/3187b710-1361-401b-a6e8-f652a1285796-kube-api-access-9kk8j\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.248988 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.248830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-hub\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.249684 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.249463 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3187b710-1361-401b-a6e8-f652a1285796-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.251363 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.251343 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-ca\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.251537 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.251513 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.251585 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.251553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-hub\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.251667 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.251646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3187b710-1361-401b-a6e8-f652a1285796-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.259452 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.259431 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kk8j\" (UniqueName: \"kubernetes.io/projected/3187b710-1361-401b-a6e8-f652a1285796-kube-api-access-9kk8j\") pod \"cluster-proxy-proxy-agent-77659ccd6-ddxln\" (UID: \"3187b710-1361-401b-a6e8-f652a1285796\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.374194 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.374103 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" Jul 15 19:30:08.502612 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:08.502581 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln"] Jul 15 19:30:08.506286 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:30:08.506260 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3187b710_1361_401b_a6e8_f652a1285796.slice/crio-c616b5edb381a3a67bb3477c59d3012df4ce9aec85bc817f77b4299a9994a485 WatchSource:0}: Error finding container c616b5edb381a3a67bb3477c59d3012df4ce9aec85bc817f77b4299a9994a485: Status 404 returned error can't find the container with id c616b5edb381a3a67bb3477c59d3012df4ce9aec85bc817f77b4299a9994a485 Jul 15 19:30:09.078769 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:09.078731 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" event={"ID":"3187b710-1361-401b-a6e8-f652a1285796","Type":"ContainerStarted","Data":"c616b5edb381a3a67bb3477c59d3012df4ce9aec85bc817f77b4299a9994a485"} Jul 15 19:30:10.364546 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:10.364515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:30:10.367007 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:10.366979 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a4df817f-0d0b-4c52-8f92-d12a27c50869-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-95x5f\" (UID: \"a4df817f-0d0b-4c52-8f92-d12a27c50869\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:30:10.645781 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:10.645699 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" Jul 15 19:30:10.773601 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:10.773566 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-95x5f"] Jul 15 19:30:10.776701 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:30:10.776675 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4df817f_0d0b_4c52_8f92_d12a27c50869.slice/crio-82f9275854b91b2b51a6b96187a3bac7d1779384cd75786b54c1b258ca81395d WatchSource:0}: Error finding container 82f9275854b91b2b51a6b96187a3bac7d1779384cd75786b54c1b258ca81395d: Status 404 returned error can't find the container with id 82f9275854b91b2b51a6b96187a3bac7d1779384cd75786b54c1b258ca81395d Jul 15 19:30:11.086348 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:11.086316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" event={"ID":"a4df817f-0d0b-4c52-8f92-d12a27c50869","Type":"ContainerStarted","Data":"82f9275854b91b2b51a6b96187a3bac7d1779384cd75786b54c1b258ca81395d"} Jul 15 19:30:13.093348 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:13.093312 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" event={"ID":"a4df817f-0d0b-4c52-8f92-d12a27c50869","Type":"ContainerStarted","Data":"161c88f6d71a840f2575ff528a8578e9e4e52fb4e3c7230f7baa1e6f8933eafb"} Jul 15 19:30:15.201251 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:15.201215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:30:15.204184 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:15.204160 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca6fa370-fa41-45c9-8b77-d0eeb61379ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w487\" (UID: \"ca6fa370-fa41-45c9-8b77-d0eeb61379ec\") " pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:30:15.224053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:15.224029 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5w487" Jul 15 19:30:15.373104 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:15.373048 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-95x5f" podStartSLOduration=35.724839368 podStartE2EDuration="37.373029057s" totalImagesPullingTime="1.648189689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:13.106777268 +0000 UTC m=+94.967981643" watchObservedRunningTime="2026-07-15 19:30:15.373029057 +0000 UTC m=+97.234233411" Jul 15 19:30:15.373637 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:15.373614 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5w487"] Jul 15 19:30:16.006361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:16.006334 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lwlwl" Jul 15 19:30:16.111439 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:16.111375 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w487" event={"ID":"ca6fa370-fa41-45c9-8b77-d0eeb61379ec","Type":"ContainerStarted","Data":"ec697d1b2bb36ea3517d7c71b82b37882de1c15f83027eb1acc3ab12b02976cf"} Jul 15 19:30:17.925917 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:17.925867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:30:17.930175 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:17.930151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6-metrics-tls\") pod \"dns-default-4gz28\" (UID: \"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6\") " pod="openshift-dns/dns-default-4gz28" Jul 15 19:30:18.203465 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:18.203441 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9zc59\"" Jul 15 19:30:18.212610 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:18.212563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4gz28" Jul 15 19:30:18.367002 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:18.366902 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4gz28"] Jul 15 19:30:18.369836 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:30:18.369783 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc122d6_cbc3_4c34_99f3_9c420f4f8ae6.slice/crio-92b26c9c5868073ad0dbc121326966d85b6862b4f8cd9eabc2dd1bb56b12564b WatchSource:0}: Error finding container 92b26c9c5868073ad0dbc121326966d85b6862b4f8cd9eabc2dd1bb56b12564b: Status 404 returned error can't find the container with id 92b26c9c5868073ad0dbc121326966d85b6862b4f8cd9eabc2dd1bb56b12564b Jul 15 19:30:19.124435 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:19.124402 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" event={"ID":"3187b710-1361-401b-a6e8-f652a1285796","Type":"ContainerStarted","Data":"325044ad2810dd8bad5355aeb2040484e49f7cf7ee4fbd7c2bcde93af755f2c5"} Jul 15 19:30:19.124879 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:19.124449 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" event={"ID":"3187b710-1361-401b-a6e8-f652a1285796","Type":"ContainerStarted","Data":"d75e945550375c290c4445dfefa070c96bf1589688f67da44e471e2b21c20832"} Jul 15 19:30:19.124879 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:19.124468 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" event={"ID":"3187b710-1361-401b-a6e8-f652a1285796","Type":"ContainerStarted","Data":"903bfd52e16b37a38dabbaeb3919b1f7ef7a74506fef7efc6426bbf27d59fcc5"} Jul 15 19:30:19.125625 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:19.125586 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gz28" event={"ID":"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6","Type":"ContainerStarted","Data":"92b26c9c5868073ad0dbc121326966d85b6862b4f8cd9eabc2dd1bb56b12564b"} Jul 15 19:30:19.141649 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:19.141607 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77659ccd6-ddxln" podStartSLOduration=1.565313262 podStartE2EDuration="11.141592742s" totalImagesPullingTime="9.57627948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:19.140626289 +0000 UTC m=+101.001830642" watchObservedRunningTime="2026-07-15 19:30:19.141592742 +0000 UTC m=+101.002797095" Jul 15 19:30:23.143053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:23.143011 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w487" event={"ID":"ca6fa370-fa41-45c9-8b77-d0eeb61379ec","Type":"ContainerStarted","Data":"a5f22ba053157b9e7358a0378efa3eb61ecbec159a6d0146f449ccdc5c924e7d"} Jul 15 19:30:23.144915 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:23.144886 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gz28" event={"ID":"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6","Type":"ContainerStarted","Data":"4ff0dac9827714d6a45cf815e1072b93b87d5bf2629340b1a77527f97010817c"} Jul 15 19:30:23.145041 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:23.144938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gz28" event={"ID":"8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6","Type":"ContainerStarted","Data":"066dd344f5f35dc83cef874a7017eae7d01284669663a9018cbeb66b52ac8ddf"} Jul 15 19:30:23.145041 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:23.144986 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4gz28" Jul 15 19:30:23.162253 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:23.162210 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4gz28" podStartSLOduration=66.279005198 podStartE2EDuration="1m10.16219337s" totalImagesPullingTime="3.883188172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:23.160845993 +0000 UTC m=+105.022050364" watchObservedRunningTime="2026-07-15 19:30:23.16219337 +0000 UTC m=+105.023397725" Jul 15 19:30:27.180013 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:27.179970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w487" event={"ID":"ca6fa370-fa41-45c9-8b77-d0eeb61379ec","Type":"ContainerStarted","Data":"71e8ae4cdda73e004fe103f029f6cdc75a58de58acd9987957cccaee7f9b1aa6"} Jul 15 19:30:30.192246 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:30.192208 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w487" event={"ID":"ca6fa370-fa41-45c9-8b77-d0eeb61379ec","Type":"ContainerStarted","Data":"d257cc45e57ac05ed4f9a5908f8df76a73011cb128eb2f1a936845377d3d3dc8"} Jul 15 19:30:30.209060 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:30.209014 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5w487" podStartSLOduration=32.949039355 podStartE2EDuration="47.208999298s" totalImagesPullingTime="14.259959943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:29:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:30.20759182 +0000 UTC m=+112.068796173" watchObservedRunningTime="2026-07-15 19:30:30.208999298 +0000 UTC m=+112.070203674" Jul 15 19:30:33.138895 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.138863 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gvcnz"] Jul 15 19:30:33.141104 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.141088 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.143325 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.143300 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 19:30:33.143567 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.143546 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 19:30:33.143654 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.143614 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 19:30:33.143654 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.143620 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 19:30:33.144241 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.144227 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 19:30:33.144321 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.144305 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x2sbw\"" Jul 15 19:30:33.144359 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.144309 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 19:30:33.150667 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.150647 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4gz28" Jul 15 19:30:33.249382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-wtmp\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-tls\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249640 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249640 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-textfile\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249640 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249558 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249640 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42da4001-b896-4ef5-a791-840a417ebcd5-metrics-client-ca\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249842 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-sys\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249905 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464xx\" (UniqueName: \"kubernetes.io/projected/42da4001-b896-4ef5-a791-840a417ebcd5-kube-api-access-464xx\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.249959 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.249943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-root\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351063 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42da4001-b896-4ef5-a791-840a417ebcd5-metrics-client-ca\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-sys\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464xx\" (UniqueName: \"kubernetes.io/projected/42da4001-b896-4ef5-a791-840a417ebcd5-kube-api-access-464xx\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-root\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-wtmp\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-tls\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351236 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351618 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-textfile\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351618 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351425 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-sys\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351618 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-wtmp\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351618 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351494 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42da4001-b896-4ef5-a791-840a417ebcd5-root\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351618 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-textfile\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.351831 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.351782 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42da4001-b896-4ef5-a791-840a417ebcd5-metrics-client-ca\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.352041 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.352013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.354028 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.353995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-tls\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.354571 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.354546 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42da4001-b896-4ef5-a791-840a417ebcd5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.365423 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.365396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-464xx\" (UniqueName: \"kubernetes.io/projected/42da4001-b896-4ef5-a791-840a417ebcd5-kube-api-access-464xx\") pod \"node-exporter-gvcnz\" (UID: \"42da4001-b896-4ef5-a791-840a417ebcd5\") " pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.450659 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:33.450627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvcnz" Jul 15 19:30:33.461410 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:30:33.461383 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42da4001_b896_4ef5_a791_840a417ebcd5.slice/crio-923d55f23c27f2efd5370eff2e19b44eedccc4fd125278150bc174207582a611 WatchSource:0}: Error finding container 923d55f23c27f2efd5370eff2e19b44eedccc4fd125278150bc174207582a611: Status 404 returned error can't find the container with id 923d55f23c27f2efd5370eff2e19b44eedccc4fd125278150bc174207582a611 Jul 15 19:30:34.208071 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:34.208039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvcnz" event={"ID":"42da4001-b896-4ef5-a791-840a417ebcd5","Type":"ContainerStarted","Data":"923d55f23c27f2efd5370eff2e19b44eedccc4fd125278150bc174207582a611"} Jul 15 19:30:35.215696 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:35.215665 2573 generic.go:356] "Generic (PLEG): container finished" podID="42da4001-b896-4ef5-a791-840a417ebcd5" containerID="95cda97ff4df9acb94eb4b826b7c5774a10b1eb58c135ab7d32196daa063695e" exitCode=0 Jul 15 19:30:35.216055 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:35.215742 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvcnz" event={"ID":"42da4001-b896-4ef5-a791-840a417ebcd5","Type":"ContainerDied","Data":"95cda97ff4df9acb94eb4b826b7c5774a10b1eb58c135ab7d32196daa063695e"} Jul 15 19:30:36.221590 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:36.221556 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvcnz" event={"ID":"42da4001-b896-4ef5-a791-840a417ebcd5","Type":"ContainerStarted","Data":"48ff4decd0ba5177b29d182da55d2fd793750ec6235a45a32169ea72492c1a99"} Jul 15 19:30:36.222032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:36.221606 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvcnz" event={"ID":"42da4001-b896-4ef5-a791-840a417ebcd5","Type":"ContainerStarted","Data":"b675d0d388ccbf700752fc6c16488da8131272cd5fb97fc6a1ca771c636c24cc"} Jul 15 19:30:36.241181 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:36.241134 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gvcnz" podStartSLOduration=1.623252154 podStartE2EDuration="3.241119238s" totalImagesPullingTime="1.545677379s" totalInitContainerRuntime="72.189705ms" isStatefulPod=true podCreationTimestamp="2026-07-15 19:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:36.240039838 +0000 UTC m=+118.101244191" watchObservedRunningTime="2026-07-15 19:30:36.241119238 +0000 UTC m=+118.102323591" Jul 15 19:30:48.361484 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:48.361440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:30:48.363963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:48.363930 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab526b6-f5f9-4a7c-ab2b-04eb387d9520-metrics-certs\") pod \"network-metrics-daemon-j5zpl\" (UID: \"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520\") " pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:30:48.603831 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:48.603778 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5v82\"" Jul 15 19:30:48.612524 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:48.612460 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j5zpl" Jul 15 19:30:48.796488 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:48.796457 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j5zpl"] Jul 15 19:30:48.799150 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:30:48.799124 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab526b6_f5f9_4a7c_ab2b_04eb387d9520.slice/crio-ed07e9cd534c6b66220d25e389a58f364c19fec5b4c86b07088bfaf2ea2157ec WatchSource:0}: Error finding container ed07e9cd534c6b66220d25e389a58f364c19fec5b4c86b07088bfaf2ea2157ec: Status 404 returned error can't find the container with id ed07e9cd534c6b66220d25e389a58f364c19fec5b4c86b07088bfaf2ea2157ec Jul 15 19:30:49.265789 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:49.265757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j5zpl" event={"ID":"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520","Type":"ContainerStarted","Data":"ed07e9cd534c6b66220d25e389a58f364c19fec5b4c86b07088bfaf2ea2157ec"} Jul 15 19:30:50.270775 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:50.270749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j5zpl" event={"ID":"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520","Type":"ContainerStarted","Data":"5aa4343b20cfb5613111d77057592c901548285e523398d95121f198b922ab00"} Jul 15 19:30:51.275451 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:51.275405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j5zpl" event={"ID":"7ab526b6-f5f9-4a7c-ab2b-04eb387d9520","Type":"ContainerStarted","Data":"61de10d9c3e322a22ec10ecbcabc306304e362290889e8ab27294f56a63930ff"} Jul 15 19:30:51.290168 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:51.290126 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j5zpl" podStartSLOduration=131.013826327 podStartE2EDuration="2m12.290112912s" totalImagesPullingTime="1.276286585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:30:51.289794208 +0000 UTC m=+133.150998579" watchObservedRunningTime="2026-07-15 19:30:51.290112912 +0000 UTC m=+133.151317264" Jul 15 19:30:58.962356 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:30:58.962316 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-72d5n_bed9667d-b54b-4056-aa7b-300f7905aa67/serve-healthcheck-canary/0.log" Jul 15 19:31:09.336157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:31:09.336122 2573 generic.go:356] "Generic (PLEG): container finished" podID="627fbf81-2e23-47ca-9a57-f577caf7e3f5" containerID="12c0db6cf3dd71fa481af7b030088238514f9906510b68ba8fea6f18dff6bb4c" exitCode=0 Jul 15 19:31:09.336698 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:31:09.336191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" event={"ID":"627fbf81-2e23-47ca-9a57-f577caf7e3f5","Type":"ContainerDied","Data":"12c0db6cf3dd71fa481af7b030088238514f9906510b68ba8fea6f18dff6bb4c"} Jul 15 19:31:09.336698 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:31:09.336565 2573 scope.go:117] "RemoveContainer" containerID="12c0db6cf3dd71fa481af7b030088238514f9906510b68ba8fea6f18dff6bb4c" Jul 15 19:31:10.343883 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:31:10.343836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qlp94" event={"ID":"627fbf81-2e23-47ca-9a57-f577caf7e3f5","Type":"ContainerStarted","Data":"8d5069f932f32ff58af11d28b1807ac6462dafb911760c8ae4b3a5b564b22682"} Jul 15 19:33:38.550397 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:33:38.550370 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:33:38.552423 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:33:38.552390 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:33:38.564275 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:33:38.564241 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:33:38.566074 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:33:38.566052 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:33:38.567145 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:33:38.567116 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 19:34:25.791076 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.791041 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq"] Jul 15 19:34:25.796432 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.796402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:25.798659 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.798633 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 19:34:25.799223 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.799202 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lcxtt\"" Jul 15 19:34:25.799223 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.799225 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 19:34:25.800718 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.800689 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq"] Jul 15 19:34:25.893204 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.893171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2197184f-a010-4992-9873-78631ec6bb3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qbbwq\" (UID: \"2197184f-a010-4992-9873-78631ec6bb3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:25.893367 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.893208 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/2197184f-a010-4992-9873-78631ec6bb3a-kube-api-access-pqdzc\") pod \"cert-manager-cainjector-559cdd8c87-qbbwq\" (UID: \"2197184f-a010-4992-9873-78631ec6bb3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:25.993503 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.993479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2197184f-a010-4992-9873-78631ec6bb3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qbbwq\" (UID: \"2197184f-a010-4992-9873-78631ec6bb3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:25.993696 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:25.993510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/2197184f-a010-4992-9873-78631ec6bb3a-kube-api-access-pqdzc\") pod \"cert-manager-cainjector-559cdd8c87-qbbwq\" (UID: \"2197184f-a010-4992-9873-78631ec6bb3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:26.001457 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:26.001424 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2197184f-a010-4992-9873-78631ec6bb3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qbbwq\" (UID: \"2197184f-a010-4992-9873-78631ec6bb3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:26.001634 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:26.001618 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdzc\" (UniqueName: \"kubernetes.io/projected/2197184f-a010-4992-9873-78631ec6bb3a-kube-api-access-pqdzc\") pod \"cert-manager-cainjector-559cdd8c87-qbbwq\" (UID: \"2197184f-a010-4992-9873-78631ec6bb3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:26.106377 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:26.106287 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" Jul 15 19:34:26.233276 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:34:26.233242 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2197184f_a010_4992_9873_78631ec6bb3a.slice/crio-fa1e2e2c86168499ed31d66b68c9e73c6e41ee66fd5a4d5a82a27677b5df9cc5 WatchSource:0}: Error finding container fa1e2e2c86168499ed31d66b68c9e73c6e41ee66fd5a4d5a82a27677b5df9cc5: Status 404 returned error can't find the container with id fa1e2e2c86168499ed31d66b68c9e73c6e41ee66fd5a4d5a82a27677b5df9cc5 Jul 15 19:34:26.233845 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:26.233719 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq"] Jul 15 19:34:26.235228 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:26.235208 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:34:27.000330 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:27.000296 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" event={"ID":"2197184f-a010-4992-9873-78631ec6bb3a","Type":"ContainerStarted","Data":"fa1e2e2c86168499ed31d66b68c9e73c6e41ee66fd5a4d5a82a27677b5df9cc5"} Jul 15 19:34:30.013263 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:30.013164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" event={"ID":"2197184f-a010-4992-9873-78631ec6bb3a","Type":"ContainerStarted","Data":"83351ac4577a27c8ddefac310dbcfb3a440c2e41953297c04c35d9f3336debe6"} Jul 15 19:34:30.028097 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:30.028037 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbbwq" podStartSLOduration=1.518315931 podStartE2EDuration="5.02801727s" totalImagesPullingTime="3.509701339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:30.026954863 +0000 UTC m=+351.888159215" watchObservedRunningTime="2026-07-15 19:34:30.02801727 +0000 UTC m=+351.889221624" Jul 15 19:34:41.374904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.374872 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r"] Jul 15 19:34:41.378045 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.378024 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.380234 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.380213 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:34:41.380684 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.380662 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8rdqb\"" Jul 15 19:34:41.380735 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.380668 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 19:34:41.384973 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.384951 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r"] Jul 15 19:34:41.397533 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.397511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ddbf45-ea28-46ec-9622-6df5e2e15771-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x5f6r\" (UID: \"f2ddbf45-ea28-46ec-9622-6df5e2e15771\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.397649 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.397558 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thgpc\" (UniqueName: \"kubernetes.io/projected/f2ddbf45-ea28-46ec-9622-6df5e2e15771-kube-api-access-thgpc\") pod \"openshift-lws-operator-fd8ccff4c-x5f6r\" (UID: \"f2ddbf45-ea28-46ec-9622-6df5e2e15771\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.498202 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.498168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thgpc\" (UniqueName: \"kubernetes.io/projected/f2ddbf45-ea28-46ec-9622-6df5e2e15771-kube-api-access-thgpc\") pod \"openshift-lws-operator-fd8ccff4c-x5f6r\" (UID: \"f2ddbf45-ea28-46ec-9622-6df5e2e15771\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.498374 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.498220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ddbf45-ea28-46ec-9622-6df5e2e15771-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x5f6r\" (UID: \"f2ddbf45-ea28-46ec-9622-6df5e2e15771\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.498538 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.498520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ddbf45-ea28-46ec-9622-6df5e2e15771-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x5f6r\" (UID: \"f2ddbf45-ea28-46ec-9622-6df5e2e15771\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.505812 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.505786 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thgpc\" (UniqueName: \"kubernetes.io/projected/f2ddbf45-ea28-46ec-9622-6df5e2e15771-kube-api-access-thgpc\") pod \"openshift-lws-operator-fd8ccff4c-x5f6r\" (UID: \"f2ddbf45-ea28-46ec-9622-6df5e2e15771\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.687893 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.687864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" Jul 15 19:34:41.814438 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:41.814410 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r"] Jul 15 19:34:41.817485 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:34:41.817459 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ddbf45_ea28_46ec_9622_6df5e2e15771.slice/crio-9622be86eae5b382e5e8beb10008ee353abb035c84dceff932533b137e8dffc6 WatchSource:0}: Error finding container 9622be86eae5b382e5e8beb10008ee353abb035c84dceff932533b137e8dffc6: Status 404 returned error can't find the container with id 9622be86eae5b382e5e8beb10008ee353abb035c84dceff932533b137e8dffc6 Jul 15 19:34:42.056650 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:42.056569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" event={"ID":"f2ddbf45-ea28-46ec-9622-6df5e2e15771","Type":"ContainerStarted","Data":"9622be86eae5b382e5e8beb10008ee353abb035c84dceff932533b137e8dffc6"} Jul 15 19:34:46.072305 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:46.072215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" event={"ID":"f2ddbf45-ea28-46ec-9622-6df5e2e15771","Type":"ContainerStarted","Data":"144d0d1a9b7db518d31205b1637a2ee561817ffbad948a240cb1188dbc680afc"} Jul 15 19:34:46.088291 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:34:46.088236 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x5f6r" podStartSLOduration=1.208130927 podStartE2EDuration="5.088218451s" totalImagesPullingTime="3.880087524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:34:46.08685291 +0000 UTC m=+367.948057264" watchObservedRunningTime="2026-07-15 19:34:46.088218451 +0000 UTC m=+367.949422804" Jul 15 19:35:01.704826 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.704778 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69"] Jul 15 19:35:01.707071 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.707054 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.709374 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.709349 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w5j8n\"" Jul 15 19:35:01.709511 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.709389 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 19:35:01.709626 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.709611 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 19:35:01.709728 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.709710 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 19:35:01.709787 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.709738 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 19:35:01.726517 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.726487 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69"] Jul 15 19:35:01.847855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.847820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6220d68c-c0f6-4b13-b355-3efa1093638c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.848032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.847874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6220d68c-c0f6-4b13-b355-3efa1093638c-tmp\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.848032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.847902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6220d68c-c0f6-4b13-b355-3efa1093638c-webhook-cert\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.848032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.847921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6220d68c-c0f6-4b13-b355-3efa1093638c-manifests\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.848032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.847938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jght7\" (UniqueName: \"kubernetes.io/projected/6220d68c-c0f6-4b13-b355-3efa1093638c-kube-api-access-jght7\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.948512 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.948475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6220d68c-c0f6-4b13-b355-3efa1093638c-webhook-cert\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.948512 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.948514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6220d68c-c0f6-4b13-b355-3efa1093638c-manifests\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.948760 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.948539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jght7\" (UniqueName: \"kubernetes.io/projected/6220d68c-c0f6-4b13-b355-3efa1093638c-kube-api-access-jght7\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.948760 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.948590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6220d68c-c0f6-4b13-b355-3efa1093638c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.948760 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.948639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6220d68c-c0f6-4b13-b355-3efa1093638c-tmp\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.949001 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.948977 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6220d68c-c0f6-4b13-b355-3efa1093638c-manifests\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.950992 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.950963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6220d68c-c0f6-4b13-b355-3efa1093638c-tmp\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.951289 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.951269 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6220d68c-c0f6-4b13-b355-3efa1093638c-webhook-cert\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.951370 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.951267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6220d68c-c0f6-4b13-b355-3efa1093638c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:01.956093 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:01.956040 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jght7\" (UniqueName: \"kubernetes.io/projected/6220d68c-c0f6-4b13-b355-3efa1093638c-kube-api-access-jght7\") pod \"opendatahub-operator-controller-manager-58dd997468-klp69\" (UID: \"6220d68c-c0f6-4b13-b355-3efa1093638c\") " pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:02.017401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:02.017367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:02.160780 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:02.160748 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69"] Jul 15 19:35:02.164174 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:02.164143 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6220d68c_c0f6_4b13_b355_3efa1093638c.slice/crio-5419a030cae60fc14cf7ddfca60ef2cefef9d72ab957503a519c413bf41c5974 WatchSource:0}: Error finding container 5419a030cae60fc14cf7ddfca60ef2cefef9d72ab957503a519c413bf41c5974: Status 404 returned error can't find the container with id 5419a030cae60fc14cf7ddfca60ef2cefef9d72ab957503a519c413bf41c5974 Jul 15 19:35:03.134698 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:03.134655 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" event={"ID":"6220d68c-c0f6-4b13-b355-3efa1093638c","Type":"ContainerStarted","Data":"5419a030cae60fc14cf7ddfca60ef2cefef9d72ab957503a519c413bf41c5974"} Jul 15 19:35:06.148065 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:06.148028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" event={"ID":"6220d68c-c0f6-4b13-b355-3efa1093638c","Type":"ContainerStarted","Data":"45cb11ca8834b12d8ab1483e4befd2b0e6bf61f59beff4d4155847d99691dc0a"} Jul 15 19:35:07.153753 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:07.153715 2573 generic.go:356] "Generic (PLEG): container finished" podID="6220d68c-c0f6-4b13-b355-3efa1093638c" containerID="45cb11ca8834b12d8ab1483e4befd2b0e6bf61f59beff4d4155847d99691dc0a" exitCode=0 Jul 15 19:35:07.154283 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:07.153825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" event={"ID":"6220d68c-c0f6-4b13-b355-3efa1093638c","Type":"ContainerDied","Data":"45cb11ca8834b12d8ab1483e4befd2b0e6bf61f59beff4d4155847d99691dc0a"} Jul 15 19:35:08.160051 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:08.160014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" event={"ID":"6220d68c-c0f6-4b13-b355-3efa1093638c","Type":"ContainerStarted","Data":"f49ec7f4588de4c459246f748ab0c25701af8ed5d565223d8532ea10ca75680d"} Jul 15 19:35:08.160426 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:08.160103 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:08.183988 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:08.183935 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" podStartSLOduration=2.436176192 podStartE2EDuration="7.183917285s" totalImagesPullingTime="3.722163459s" totalInitContainerRuntime="1.025577634s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:08.182528573 +0000 UTC m=+390.043732948" watchObservedRunningTime="2026-07-15 19:35:08.183917285 +0000 UTC m=+390.045121716" Jul 15 19:35:19.166434 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:19.166406 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58dd997468-klp69" Jul 15 19:35:21.735384 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:21.735340 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lnbsw"] Jul 15 19:35:21.743685 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:21.743664 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:21.746849 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:21.746830 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 19:35:21.747556 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:21.747531 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dg5zm\"" Jul 15 19:35:21.762924 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:21.762905 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lnbsw"] Jul 15 19:35:21.905444 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:21.905409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-cert\") pod \"odh-model-controller-7998c67dc4-lnbsw\" (UID: \"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865\") " pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:21.905620 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:21.905465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2c8\" (UniqueName: \"kubernetes.io/projected/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-kube-api-access-4r2c8\") pod \"odh-model-controller-7998c67dc4-lnbsw\" (UID: \"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865\") " pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:22.006736 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.006652 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-cert\") pod \"odh-model-controller-7998c67dc4-lnbsw\" (UID: \"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865\") " pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:22.006736 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.006726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2c8\" (UniqueName: \"kubernetes.io/projected/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-kube-api-access-4r2c8\") pod \"odh-model-controller-7998c67dc4-lnbsw\" (UID: \"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865\") " pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:22.006942 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:35:22.006830 2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 19:35:22.006942 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:35:22.006895 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-cert podName:277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865 nodeName:}" failed. No retries permitted until 2026-07-15 19:35:22.506877343 +0000 UTC m=+404.368081675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-cert") pod "odh-model-controller-7998c67dc4-lnbsw" (UID: "277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865") : secret "odh-model-controller-webhook-cert" not found Jul 15 19:35:22.024895 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.024859 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2c8\" (UniqueName: \"kubernetes.io/projected/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-kube-api-access-4r2c8\") pod \"odh-model-controller-7998c67dc4-lnbsw\" (UID: \"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865\") " pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:22.460447 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.460417 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7zhk8"] Jul 15 19:35:22.463570 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.463556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.465415 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.465391 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 19:35:22.465538 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.465458 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4r5js\"" Jul 15 19:35:22.470971 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.470953 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7zhk8"] Jul 15 19:35:22.510573 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.510541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-cert\") pod \"odh-model-controller-7998c67dc4-lnbsw\" (UID: \"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865\") " pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:22.513074 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.513044 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865-cert\") pod \"odh-model-controller-7998c67dc4-lnbsw\" (UID: \"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865\") " pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:22.611593 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.611545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gw6\" (UniqueName: \"kubernetes.io/projected/f1997ede-bf93-4836-b5e4-73337a6d1113-kube-api-access-k5gw6\") pod \"model-serving-api-54d89bd5fc-7zhk8\" (UID: \"f1997ede-bf93-4836-b5e4-73337a6d1113\") " pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.611760 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.611656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1997ede-bf93-4836-b5e4-73337a6d1113-tls-certs\") pod \"model-serving-api-54d89bd5fc-7zhk8\" (UID: \"f1997ede-bf93-4836-b5e4-73337a6d1113\") " pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.653623 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.653596 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:22.712679 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.712615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gw6\" (UniqueName: \"kubernetes.io/projected/f1997ede-bf93-4836-b5e4-73337a6d1113-kube-api-access-k5gw6\") pod \"model-serving-api-54d89bd5fc-7zhk8\" (UID: \"f1997ede-bf93-4836-b5e4-73337a6d1113\") " pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.712679 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.712667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1997ede-bf93-4836-b5e4-73337a6d1113-tls-certs\") pod \"model-serving-api-54d89bd5fc-7zhk8\" (UID: \"f1997ede-bf93-4836-b5e4-73337a6d1113\") " pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.715319 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.715259 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1997ede-bf93-4836-b5e4-73337a6d1113-tls-certs\") pod \"model-serving-api-54d89bd5fc-7zhk8\" (UID: \"f1997ede-bf93-4836-b5e4-73337a6d1113\") " pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.721401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.721374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gw6\" (UniqueName: \"kubernetes.io/projected/f1997ede-bf93-4836-b5e4-73337a6d1113-kube-api-access-k5gw6\") pod \"model-serving-api-54d89bd5fc-7zhk8\" (UID: \"f1997ede-bf93-4836-b5e4-73337a6d1113\") " pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.774314 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.774289 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:22.781045 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.781026 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lnbsw"] Jul 15 19:35:22.784531 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:22.784507 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277e1c02_9dc1_4e0f_8e01_d4e4e2fbe865.slice/crio-726f4b7ac9441c8b32b3d940706f34f547c56d04bd837029762a782d5d58971a WatchSource:0}: Error finding container 726f4b7ac9441c8b32b3d940706f34f547c56d04bd837029762a782d5d58971a: Status 404 returned error can't find the container with id 726f4b7ac9441c8b32b3d940706f34f547c56d04bd837029762a782d5d58971a Jul 15 19:35:22.903073 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:22.903041 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7zhk8"] Jul 15 19:35:22.907454 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:22.907410 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1997ede_bf93_4836_b5e4_73337a6d1113.slice/crio-8ab5f394d571651584312d2fad10018e11f5a28d20ab945617db3c5f679a170d WatchSource:0}: Error finding container 8ab5f394d571651584312d2fad10018e11f5a28d20ab945617db3c5f679a170d: Status 404 returned error can't find the container with id 8ab5f394d571651584312d2fad10018e11f5a28d20ab945617db3c5f679a170d Jul 15 19:35:23.217241 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:23.217201 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" event={"ID":"f1997ede-bf93-4836-b5e4-73337a6d1113","Type":"ContainerStarted","Data":"8ab5f394d571651584312d2fad10018e11f5a28d20ab945617db3c5f679a170d"} Jul 15 19:35:23.218278 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:23.218254 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" event={"ID":"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865","Type":"ContainerStarted","Data":"726f4b7ac9441c8b32b3d940706f34f547c56d04bd837029762a782d5d58971a"} Jul 15 19:35:26.236746 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.236708 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" event={"ID":"277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865","Type":"ContainerStarted","Data":"ceb5023501e8cd51fc9da89b87a91c2b0e7045a939126bf991125ce163f22f75"} Jul 15 19:35:26.237166 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.236781 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:26.238257 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.238238 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" event={"ID":"f1997ede-bf93-4836-b5e4-73337a6d1113","Type":"ContainerStarted","Data":"6a0a725b91e1ff5695048aa3744cb197d99ba1a656178000137b77db1357469d"} Jul 15 19:35:26.238341 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.238293 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:26.253847 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.253777 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" podStartSLOduration=2.227335531 podStartE2EDuration="5.253763905s" totalImagesPullingTime="3.026428374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:26.253230554 +0000 UTC m=+408.114434927" watchObservedRunningTime="2026-07-15 19:35:26.253763905 +0000 UTC m=+408.114968258" Jul 15 19:35:26.268664 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.268613 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" podStartSLOduration=1.408674508 podStartE2EDuration="4.268599029s" totalImagesPullingTime="2.859924521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:26.2672031 +0000 UTC m=+408.128407453" watchObservedRunningTime="2026-07-15 19:35:26.268599029 +0000 UTC m=+408.129803383" Jul 15 19:35:26.876929 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.876891 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9"] Jul 15 19:35:26.880084 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.880062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:26.882178 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.882125 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 19:35:26.882340 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.882321 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 19:35:26.882453 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.882435 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 19:35:26.882501 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.882441 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 19:35:26.890788 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:26.890767 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9"] Jul 15 19:35:27.047496 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.047465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tmp\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.047648 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.047503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh69n\" (UniqueName: \"kubernetes.io/projected/0378c7a5-591a-45ee-aa34-2d7336d7e01d-kube-api-access-sh69n\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.047648 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.047554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tls-certs\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.148185 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.148090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tmp\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.148185 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.148132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh69n\" (UniqueName: \"kubernetes.io/projected/0378c7a5-591a-45ee-aa34-2d7336d7e01d-kube-api-access-sh69n\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.148185 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.148166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tls-certs\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.148415 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:35:27.148285 2573 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 15 19:35:27.148415 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:35:27.148349 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tls-certs podName:0378c7a5-591a-45ee-aa34-2d7336d7e01d nodeName:}" failed. No retries permitted until 2026-07-15 19:35:27.648329802 +0000 UTC m=+409.509534133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tls-certs") pod "kube-auth-proxy-79b66bc9f5-wz2w9" (UID: "0378c7a5-591a-45ee-aa34-2d7336d7e01d") : secret "kube-auth-proxy-tls" not found Jul 15 19:35:27.150520 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.150492 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tmp\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.158341 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.158313 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh69n\" (UniqueName: \"kubernetes.io/projected/0378c7a5-591a-45ee-aa34-2d7336d7e01d-kube-api-access-sh69n\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.651498 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.651464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tls-certs\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.653995 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.653974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0378c7a5-591a-45ee-aa34-2d7336d7e01d-tls-certs\") pod \"kube-auth-proxy-79b66bc9f5-wz2w9\" (UID: \"0378c7a5-591a-45ee-aa34-2d7336d7e01d\") " pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.789028 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.788997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" Jul 15 19:35:27.935798 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.935750 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9"] Jul 15 19:35:27.939420 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:27.939397 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0378c7a5_591a_45ee_aa34_2d7336d7e01d.slice/crio-725f6c2334b80a82d898f2b8c54e87af9e3f67106093a0a369e21360997b9b86 WatchSource:0}: Error finding container 725f6c2334b80a82d898f2b8c54e87af9e3f67106093a0a369e21360997b9b86: Status 404 returned error can't find the container with id 725f6c2334b80a82d898f2b8c54e87af9e3f67106093a0a369e21360997b9b86 Jul 15 19:35:27.998676 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:27.998645 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl"] Jul 15 19:35:28.003045 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.003023 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.005160 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.005138 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 19:35:28.005258 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.005179 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 19:35:28.005458 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.005441 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 19:35:28.005552 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.005486 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pbpdz\"" Jul 15 19:35:28.005552 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.005479 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 19:35:28.012451 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.012425 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl"] Jul 15 19:35:28.156896 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.156868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-cacerts\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.157057 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.156925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcc82\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-kube-api-access-gcc82\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.157057 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.157003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-local-certs\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.157144 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.157068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.157144 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.157108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-token\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.157222 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.157138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.157222 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.157170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.247514 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.247428 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" event={"ID":"0378c7a5-591a-45ee-aa34-2d7336d7e01d","Type":"ContainerStarted","Data":"725f6c2334b80a82d898f2b8c54e87af9e3f67106093a0a369e21360997b9b86"} Jul 15 19:35:28.257718 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.257696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-cacerts\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.257855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.257736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcc82\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-kube-api-access-gcc82\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.257855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.257760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-local-certs\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.257855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.257780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.257855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.257825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-token\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.257855 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.257849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.258092 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.257869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.258662 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.258556 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.260339 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.260318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-local-certs\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.260429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.260347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-cacerts\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.260429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.260360 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.260429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.260364 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.265483 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.265456 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-token\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.265713 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.265693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcc82\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-kube-api-access-gcc82\") pod \"istiod-openshift-gateway-7574459656-jdjjl\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.312667 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.312637 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:28.445500 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:28.445474 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl"] Jul 15 19:35:28.447818 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:28.447775 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8dcb84_78a5_4ea5_a8e6_2bf8b111852a.slice/crio-84a3d7a4163bd9d92b7d884f8eb0abd08f52c0d84a09341e857d01d0b8fe56ca WatchSource:0}: Error finding container 84a3d7a4163bd9d92b7d884f8eb0abd08f52c0d84a09341e857d01d0b8fe56ca: Status 404 returned error can't find the container with id 84a3d7a4163bd9d92b7d884f8eb0abd08f52c0d84a09341e857d01d0b8fe56ca Jul 15 19:35:29.254679 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:29.254630 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" event={"ID":"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a","Type":"ContainerStarted","Data":"84a3d7a4163bd9d92b7d884f8eb0abd08f52c0d84a09341e857d01d0b8fe56ca"} Jul 15 19:35:32.081679 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.081647 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg"] Jul 15 19:35:32.092583 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.092541 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6"] Jul 15 19:35:32.092742 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.092685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.095397 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.095372 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-s2gnc\"" Jul 15 19:35:32.095734 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.095709 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 19:35:32.097263 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.097241 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg"] Jul 15 19:35:32.097437 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.097346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.102064 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.102040 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dfj6b\"" Jul 15 19:35:32.102360 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.102191 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 19:35:32.105514 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.105493 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6"] Jul 15 19:35:32.192356 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.192335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9cl\" (UniqueName: \"kubernetes.io/projected/c65b3e58-47cf-4ded-87bd-d90e419b4d92-kube-api-access-ct9cl\") pod \"llmisvc-controller-manager-777c6bd7d5-28qt6\" (UID: \"c65b3e58-47cf-4ded-87bd-d90e419b4d92\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.192477 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.192373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxl7k\" (UniqueName: \"kubernetes.io/projected/b68bcc89-e760-4c37-a111-aa9c885dedbd-kube-api-access-nxl7k\") pod \"kserve-controller-manager-d6f4b65b8-4kfsg\" (UID: \"b68bcc89-e760-4c37-a111-aa9c885dedbd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.192477 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.192407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b68bcc89-e760-4c37-a111-aa9c885dedbd-cert\") pod \"kserve-controller-manager-d6f4b65b8-4kfsg\" (UID: \"b68bcc89-e760-4c37-a111-aa9c885dedbd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.192477 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.192466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c65b3e58-47cf-4ded-87bd-d90e419b4d92-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-28qt6\" (UID: \"c65b3e58-47cf-4ded-87bd-d90e419b4d92\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.210319 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.210278 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:35:32.210426 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.210361 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:35:32.293692 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.293663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9cl\" (UniqueName: \"kubernetes.io/projected/c65b3e58-47cf-4ded-87bd-d90e419b4d92-kube-api-access-ct9cl\") pod \"llmisvc-controller-manager-777c6bd7d5-28qt6\" (UID: \"c65b3e58-47cf-4ded-87bd-d90e419b4d92\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.293860 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.293718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxl7k\" (UniqueName: \"kubernetes.io/projected/b68bcc89-e760-4c37-a111-aa9c885dedbd-kube-api-access-nxl7k\") pod \"kserve-controller-manager-d6f4b65b8-4kfsg\" (UID: \"b68bcc89-e760-4c37-a111-aa9c885dedbd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.293860 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.293756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b68bcc89-e760-4c37-a111-aa9c885dedbd-cert\") pod \"kserve-controller-manager-d6f4b65b8-4kfsg\" (UID: \"b68bcc89-e760-4c37-a111-aa9c885dedbd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.293860 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.293787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c65b3e58-47cf-4ded-87bd-d90e419b4d92-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-28qt6\" (UID: \"c65b3e58-47cf-4ded-87bd-d90e419b4d92\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.296429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.296329 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b68bcc89-e760-4c37-a111-aa9c885dedbd-cert\") pod \"kserve-controller-manager-d6f4b65b8-4kfsg\" (UID: \"b68bcc89-e760-4c37-a111-aa9c885dedbd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.296551 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.296432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c65b3e58-47cf-4ded-87bd-d90e419b4d92-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-28qt6\" (UID: \"c65b3e58-47cf-4ded-87bd-d90e419b4d92\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.305611 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.305586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxl7k\" (UniqueName: \"kubernetes.io/projected/b68bcc89-e760-4c37-a111-aa9c885dedbd-kube-api-access-nxl7k\") pod \"kserve-controller-manager-d6f4b65b8-4kfsg\" (UID: \"b68bcc89-e760-4c37-a111-aa9c885dedbd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.306300 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.306276 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9cl\" (UniqueName: \"kubernetes.io/projected/c65b3e58-47cf-4ded-87bd-d90e419b4d92-kube-api-access-ct9cl\") pod \"llmisvc-controller-manager-777c6bd7d5-28qt6\" (UID: \"c65b3e58-47cf-4ded-87bd-d90e419b4d92\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.410107 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.410073 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:32.417327 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.417306 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:32.578979 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.578887 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg"] Jul 15 19:35:32.583518 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:32.583489 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68bcc89_e760_4c37_a111_aa9c885dedbd.slice/crio-7854d4aaae541fb7bcbce8e71a3dab46930f0a018d8736c4e67945f7f45216fa WatchSource:0}: Error finding container 7854d4aaae541fb7bcbce8e71a3dab46930f0a018d8736c4e67945f7f45216fa: Status 404 returned error can't find the container with id 7854d4aaae541fb7bcbce8e71a3dab46930f0a018d8736c4e67945f7f45216fa Jul 15 19:35:32.604398 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:32.604369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6"] Jul 15 19:35:32.608372 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:32.608345 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc65b3e58_47cf_4ded_87bd_d90e419b4d92.slice/crio-6c8b24aaf390204d6244cdaba2db00e0152959fafe77a79f036590ae8d8fc6e4 WatchSource:0}: Error finding container 6c8b24aaf390204d6244cdaba2db00e0152959fafe77a79f036590ae8d8fc6e4: Status 404 returned error can't find the container with id 6c8b24aaf390204d6244cdaba2db00e0152959fafe77a79f036590ae8d8fc6e4 Jul 15 19:35:33.276117 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.276078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" event={"ID":"b68bcc89-e760-4c37-a111-aa9c885dedbd","Type":"ContainerStarted","Data":"7854d4aaae541fb7bcbce8e71a3dab46930f0a018d8736c4e67945f7f45216fa"} Jul 15 19:35:33.278611 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.278057 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" event={"ID":"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a","Type":"ContainerStarted","Data":"fe1d93303f5875851e3c83752e121eecc5ca757a4b013772dfc3d8cf5549bb48"} Jul 15 19:35:33.278611 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.278298 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:33.280172 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.280142 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-jdjjl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:35:33.280298 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.280199 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" podUID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:35:33.280298 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.280226 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" event={"ID":"0378c7a5-591a-45ee-aa34-2d7336d7e01d","Type":"ContainerStarted","Data":"54c6dc246285b0ed04b691a063c535076f48dfc2ace7bc9a294ae61ec9cfa817"} Jul 15 19:35:33.281912 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.281884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" event={"ID":"c65b3e58-47cf-4ded-87bd-d90e419b4d92","Type":"ContainerStarted","Data":"6c8b24aaf390204d6244cdaba2db00e0152959fafe77a79f036590ae8d8fc6e4"} Jul 15 19:35:33.334318 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.334258 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" podStartSLOduration=2.573918673 podStartE2EDuration="6.334235504s" totalImagesPullingTime="3.760316831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:33.333874745 +0000 UTC m=+415.195079118" watchObservedRunningTime="2026-07-15 19:35:33.334235504 +0000 UTC m=+415.195439840" Jul 15 19:35:33.383909 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.383831 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79b66bc9f5-wz2w9" podStartSLOduration=3.165878361 podStartE2EDuration="7.383791602s" totalImagesPullingTime="4.217913241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:33.382222337 +0000 UTC m=+415.243426709" watchObservedRunningTime="2026-07-15 19:35:33.383791602 +0000 UTC m=+415.244995956" Jul 15 19:35:33.906274 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.906232 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2"] Jul 15 19:35:33.909815 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.909774 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:33.917441 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.917413 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rcklq\"" Jul 15 19:35:33.943520 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:33.943442 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2"] Jul 15 19:35:34.012392 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012574 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvng\" (UniqueName: \"kubernetes.io/projected/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-kube-api-access-cpvng\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012574 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012574 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012574 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012751 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012578 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012751 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012751 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.012751 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.012678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.113829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.113765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114015 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.113851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114015 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.113894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114015 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.113935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114015 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114245 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvng\" (UniqueName: \"kubernetes.io/projected/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-kube-api-access-cpvng\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114245 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114245 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114245 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114245 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114520 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114584 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114556 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.114638 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.115000 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.114972 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.116875 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.116827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.117193 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.117171 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.124311 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.124284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvng\" (UniqueName: \"kubernetes.io/projected/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-kube-api-access-cpvng\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.125057 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.124990 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b2785f-dd7a-4ba9-a256-f54fb00ce82a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6789c8m2\" (UID: \"11b2785f-dd7a-4ba9-a256-f54fb00ce82a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.224865 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.224759 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:34.292180 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.291750 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:34.407705 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:34.407674 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2"] Jul 15 19:35:34.411087 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:35:34.411057 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b2785f_dd7a_4ba9_a256_f54fb00ce82a.slice/crio-d90ffac7b1f7fb4714c902332c330be4192ff84bee58c0b38069437b7f53c8e1 WatchSource:0}: Error finding container d90ffac7b1f7fb4714c902332c330be4192ff84bee58c0b38069437b7f53c8e1: Status 404 returned error can't find the container with id d90ffac7b1f7fb4714c902332c330be4192ff84bee58c0b38069437b7f53c8e1 Jul 15 19:35:35.300046 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:35.300006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" event={"ID":"11b2785f-dd7a-4ba9-a256-f54fb00ce82a","Type":"ContainerStarted","Data":"d90ffac7b1f7fb4714c902332c330be4192ff84bee58c0b38069437b7f53c8e1"} Jul 15 19:35:35.416285 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:35.416243 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl"] Jul 15 19:35:36.304658 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:36.304612 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" podUID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" containerName="discovery" containerID="cri-o://fe1d93303f5875851e3c83752e121eecc5ca757a4b013772dfc3d8cf5549bb48" gracePeriod=30 Jul 15 19:35:37.244923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.244893 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lnbsw" Jul 15 19:35:37.247057 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.247034 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7zhk8" Jul 15 19:35:37.313029 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.312982 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" event={"ID":"b68bcc89-e760-4c37-a111-aa9c885dedbd","Type":"ContainerStarted","Data":"5516fa6e1ca0489a753ed59612a129f82395f173c69941af82ff307c7167b5c4"} Jul 15 19:35:37.313465 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.313106 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:35:37.315357 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.315330 2573 generic.go:356] "Generic (PLEG): container finished" podID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" containerID="fe1d93303f5875851e3c83752e121eecc5ca757a4b013772dfc3d8cf5549bb48" exitCode=0 Jul 15 19:35:37.315506 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.315405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" event={"ID":"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a","Type":"ContainerDied","Data":"fe1d93303f5875851e3c83752e121eecc5ca757a4b013772dfc3d8cf5549bb48"} Jul 15 19:35:37.329539 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.329492 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" podStartSLOduration=1.333128799 podStartE2EDuration="5.329476757s" totalImagesPullingTime="3.996347958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:37.327526477 +0000 UTC m=+419.188730862" watchObservedRunningTime="2026-07-15 19:35:37.329476757 +0000 UTC m=+419.190681109" Jul 15 19:35:37.686788 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.686760 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:37.707221 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.707183 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:35:37.707310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.707258 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:35:37.707310 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.707297 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:35:37.747688 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.747658 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-token\") pod \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " Jul 15 19:35:37.747847 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.747717 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-cacerts\") pod \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " Jul 15 19:35:37.747894 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.747854 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-dns-cert\") pod \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " Jul 15 19:35:37.747941 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.747904 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-local-certs\") pod \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " Jul 15 19:35:37.747994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.747956 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-ca-configmap\") pod \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " Jul 15 19:35:37.747994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.747983 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-kubeconfig\") pod \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " Jul 15 19:35:37.748101 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.748042 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcc82\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-kube-api-access-gcc82\") pod \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\" (UID: \"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a\") " Jul 15 19:35:37.748654 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.748586 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" (UID: "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:35:37.750449 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.750399 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-cacerts" (OuterVolumeSpecName: "cacerts") pod "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" (UID: "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:35:37.750592 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.750571 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" (UID: "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:35:37.750691 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.750670 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" (UID: "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:35:37.750790 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.750768 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" (UID: "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:35:37.750877 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.750795 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-local-certs" (OuterVolumeSpecName: "local-certs") pod "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" (UID: "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:35:37.750915 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.750902 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-kube-api-access-gcc82" (OuterVolumeSpecName: "kube-api-access-gcc82") pod "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" (UID: "5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a"). InnerVolumeSpecName "kube-api-access-gcc82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:35:37.849620 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.849583 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:35:37.849620 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.849609 2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-cacerts\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:35:37.849620 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.849618 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-dns-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:35:37.849620 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.849627 2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-local-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:35:37.849963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.849636 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-csr-ca-configmap\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:35:37.849963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.849646 2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-istio-kubeconfig\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:35:37.849963 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:37.849656 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcc82\" (UniqueName: \"kubernetes.io/projected/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a-kube-api-access-gcc82\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:35:38.321880 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.321836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" event={"ID":"11b2785f-dd7a-4ba9-a256-f54fb00ce82a","Type":"ContainerStarted","Data":"2e78270c26059f00469987bae77d189bb45fa3be24caaaaaa8c936af5a606982"} Jul 15 19:35:38.323251 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.323231 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" Jul 15 19:35:38.323367 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.323283 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" event={"ID":"5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a","Type":"ContainerDied","Data":"84a3d7a4163bd9d92b7d884f8eb0abd08f52c0d84a09341e857d01d0b8fe56ca"} Jul 15 19:35:38.323367 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.323336 2573 scope.go:117] "RemoveContainer" containerID="fe1d93303f5875851e3c83752e121eecc5ca757a4b013772dfc3d8cf5549bb48" Jul 15 19:35:38.324994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.324958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" event={"ID":"c65b3e58-47cf-4ded-87bd-d90e419b4d92","Type":"ContainerStarted","Data":"537d5bd5b86df643ee1d8f3d3fac3351c73b9c098e2126c53f496aed2a78c524"} Jul 15 19:35:38.351262 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.351212 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" podStartSLOduration=2.057853571 podStartE2EDuration="5.35119614s" totalImagesPullingTime="3.293342569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:38.350357692 +0000 UTC m=+420.211562057" watchObservedRunningTime="2026-07-15 19:35:38.35119614 +0000 UTC m=+420.212400493" Jul 15 19:35:38.364717 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.364691 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl"] Jul 15 19:35:38.368210 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.368184 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl"] Jul 15 19:35:38.382723 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.382662 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" podStartSLOduration=2.407788171 podStartE2EDuration="6.382645448s" totalImagesPullingTime="3.974857277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:35:38.381239325 +0000 UTC m=+420.242443701" watchObservedRunningTime="2026-07-15 19:35:38.382645448 +0000 UTC m=+420.243849801" Jul 15 19:35:38.696352 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:38.696317 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" path="/var/lib/kubelet/pods/5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a/volumes" Jul 15 19:35:39.225688 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:39.225649 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:39.229966 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:39.229934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:39.329394 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:39.329368 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:39.329864 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:39.329417 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:35:39.330338 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:39.330316 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6789c8m2" Jul 15 19:35:42.291440 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:42.291409 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-jdjjl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 19:35:42.291845 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:35:42.291465 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-jdjjl" podUID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.22:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 19:36:08.330711 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:08.330635 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4kfsg" Jul 15 19:36:10.336185 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:10.336154 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28qt6" Jul 15 19:36:22.630155 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.630121 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhq6n"] Jul 15 19:36:22.630548 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.630446 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" containerName="discovery" Jul 15 19:36:22.630548 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.630456 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" containerName="discovery" Jul 15 19:36:22.630548 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.630520 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8dcb84-78a5-4ea5-a8e6-2bf8b111852a" containerName="discovery" Jul 15 19:36:22.641572 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.641542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" Jul 15 19:36:22.642405 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.642377 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhq6n"] Jul 15 19:36:22.643760 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.643739 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:36:22.644607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.644586 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:36:22.644693 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.644591 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-df4wt\"" Jul 15 19:36:22.715469 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.715437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ndv\" (UniqueName: \"kubernetes.io/projected/d6a8cd5e-744c-4041-bb96-b144aed53c59-kube-api-access-j2ndv\") pod \"kuadrant-operator-catalog-jhq6n\" (UID: \"d6a8cd5e-744c-4041-bb96-b144aed53c59\") " pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" Jul 15 19:36:22.816509 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.816481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ndv\" (UniqueName: \"kubernetes.io/projected/d6a8cd5e-744c-4041-bb96-b144aed53c59-kube-api-access-j2ndv\") pod \"kuadrant-operator-catalog-jhq6n\" (UID: \"d6a8cd5e-744c-4041-bb96-b144aed53c59\") " pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" Jul 15 19:36:22.823841 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.823818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ndv\" (UniqueName: \"kubernetes.io/projected/d6a8cd5e-744c-4041-bb96-b144aed53c59-kube-api-access-j2ndv\") pod \"kuadrant-operator-catalog-jhq6n\" (UID: \"d6a8cd5e-744c-4041-bb96-b144aed53c59\") " pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" Jul 15 19:36:22.951420 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:22.951388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" Jul 15 19:36:23.076732 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.076706 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhq6n"] Jul 15 19:36:23.081912 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:36:23.081879 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a8cd5e_744c_4041_bb96_b144aed53c59.slice/crio-af2064b91b10139bb03076a481ae9cfebfa14ee894403f7ea64556336e53c0a3 WatchSource:0}: Error finding container af2064b91b10139bb03076a481ae9cfebfa14ee894403f7ea64556336e53c0a3: Status 404 returned error can't find the container with id af2064b91b10139bb03076a481ae9cfebfa14ee894403f7ea64556336e53c0a3 Jul 15 19:36:23.308207 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.308130 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhq6n"] Jul 15 19:36:23.323276 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.323250 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2pbn2"] Jul 15 19:36:23.326349 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.326320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:23.337794 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.335624 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2pbn2"] Jul 15 19:36:23.421162 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.421108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88jm\" (UniqueName: \"kubernetes.io/projected/1332a400-9cdd-4e7c-b86c-f9bda5e05bd4-kube-api-access-n88jm\") pod \"kuadrant-operator-catalog-2pbn2\" (UID: \"1332a400-9cdd-4e7c-b86c-f9bda5e05bd4\") " pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:23.514201 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.514164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" event={"ID":"d6a8cd5e-744c-4041-bb96-b144aed53c59","Type":"ContainerStarted","Data":"af2064b91b10139bb03076a481ae9cfebfa14ee894403f7ea64556336e53c0a3"} Jul 15 19:36:23.521703 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.521680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88jm\" (UniqueName: \"kubernetes.io/projected/1332a400-9cdd-4e7c-b86c-f9bda5e05bd4-kube-api-access-n88jm\") pod \"kuadrant-operator-catalog-2pbn2\" (UID: \"1332a400-9cdd-4e7c-b86c-f9bda5e05bd4\") " pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:23.528763 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.528739 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88jm\" (UniqueName: \"kubernetes.io/projected/1332a400-9cdd-4e7c-b86c-f9bda5e05bd4-kube-api-access-n88jm\") pod \"kuadrant-operator-catalog-2pbn2\" (UID: \"1332a400-9cdd-4e7c-b86c-f9bda5e05bd4\") " pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:23.642070 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.641997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:23.877702 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:23.877673 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2pbn2"] Jul 15 19:36:23.892692 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:36:23.892627 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1332a400_9cdd_4e7c_b86c_f9bda5e05bd4.slice/crio-d3217dd9ecd254e15cc1aec6009e28896b36af1e7cb0159e3ab99cba290d89d9 WatchSource:0}: Error finding container d3217dd9ecd254e15cc1aec6009e28896b36af1e7cb0159e3ab99cba290d89d9: Status 404 returned error can't find the container with id d3217dd9ecd254e15cc1aec6009e28896b36af1e7cb0159e3ab99cba290d89d9 Jul 15 19:36:24.522481 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:24.522403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" event={"ID":"1332a400-9cdd-4e7c-b86c-f9bda5e05bd4","Type":"ContainerStarted","Data":"d3217dd9ecd254e15cc1aec6009e28896b36af1e7cb0159e3ab99cba290d89d9"} Jul 15 19:36:25.530362 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.530317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" event={"ID":"1332a400-9cdd-4e7c-b86c-f9bda5e05bd4","Type":"ContainerStarted","Data":"0c944ab2df67609309608b3fe97c53dced2b51341ee06ae9b19af8901b23e474"} Jul 15 19:36:25.532350 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.532281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" event={"ID":"d6a8cd5e-744c-4041-bb96-b144aed53c59","Type":"ContainerStarted","Data":"fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587"} Jul 15 19:36:25.532519 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.532376 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" podUID="d6a8cd5e-744c-4041-bb96-b144aed53c59" containerName="registry-server" containerID="cri-o://fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587" gracePeriod=2 Jul 15 19:36:25.547278 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.547196 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" podStartSLOduration=1.14294112 podStartE2EDuration="2.547182259s" totalImagesPullingTime="1.404241139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:36:25.545595125 +0000 UTC m=+467.406799477" watchObservedRunningTime="2026-07-15 19:36:25.547182259 +0000 UTC m=+467.408386612" Jul 15 19:36:25.559723 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.559680 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" podStartSLOduration=1.3389637699999999 podStartE2EDuration="3.5596643s" totalImagesPullingTime="2.22070053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:36:25.558949938 +0000 UTC m=+467.420154324" watchObservedRunningTime="2026-07-15 19:36:25.5596643 +0000 UTC m=+467.420868653" Jul 15 19:36:25.781512 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.781491 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jhq6n_d6a8cd5e-744c-4041-bb96-b144aed53c59/registry-server/0.log" Jul 15 19:36:25.783882 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.783868 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" Jul 15 19:36:25.945071 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.945036 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2ndv\" (UniqueName: \"kubernetes.io/projected/d6a8cd5e-744c-4041-bb96-b144aed53c59-kube-api-access-j2ndv\") pod \"d6a8cd5e-744c-4041-bb96-b144aed53c59\" (UID: \"d6a8cd5e-744c-4041-bb96-b144aed53c59\") " Jul 15 19:36:25.947311 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:25.947287 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a8cd5e-744c-4041-bb96-b144aed53c59-kube-api-access-j2ndv" (OuterVolumeSpecName: "kube-api-access-j2ndv") pod "d6a8cd5e-744c-4041-bb96-b144aed53c59" (UID: "d6a8cd5e-744c-4041-bb96-b144aed53c59"). InnerVolumeSpecName "kube-api-access-j2ndv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:36:26.045553 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.045521 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2ndv\" (UniqueName: \"kubernetes.io/projected/d6a8cd5e-744c-4041-bb96-b144aed53c59-kube-api-access-j2ndv\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:36:26.537716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.537692 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jhq6n_d6a8cd5e-744c-4041-bb96-b144aed53c59/registry-server/0.log" Jul 15 19:36:26.538109 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.537731 2573 generic.go:356] "Generic (PLEG): container finished" podID="d6a8cd5e-744c-4041-bb96-b144aed53c59" containerID="fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587" exitCode=1 Jul 15 19:36:26.538109 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.537783 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" Jul 15 19:36:26.538109 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.537840 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" event={"ID":"d6a8cd5e-744c-4041-bb96-b144aed53c59","Type":"ContainerDied","Data":"fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587"} Jul 15 19:36:26.538109 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.537894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jhq6n" event={"ID":"d6a8cd5e-744c-4041-bb96-b144aed53c59","Type":"ContainerDied","Data":"af2064b91b10139bb03076a481ae9cfebfa14ee894403f7ea64556336e53c0a3"} Jul 15 19:36:26.538109 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.537917 2573 scope.go:117] "RemoveContainer" containerID="fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587" Jul 15 19:36:26.547194 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.547179 2573 scope.go:117] "RemoveContainer" containerID="fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587" Jul 15 19:36:26.547463 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:36:26.547443 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587\": container with ID starting with fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587 not found: ID does not exist" containerID="fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587" Jul 15 19:36:26.547542 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.547471 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587"} err="failed to get container status \"fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587\": rpc error: code = NotFound desc = could not find container \"fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587\": container with ID starting with fdc7384241dc8d9386a7116f6fd2601a1770bd36eb0d670684011df958ef8587 not found: ID does not exist" Jul 15 19:36:26.556799 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.556778 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhq6n"] Jul 15 19:36:26.562103 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.562077 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhq6n"] Jul 15 19:36:26.694718 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:26.694677 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a8cd5e-744c-4041-bb96-b144aed53c59" path="/var/lib/kubelet/pods/d6a8cd5e-744c-4041-bb96-b144aed53c59/volumes" Jul 15 19:36:33.642392 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:33.642353 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:33.642861 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:33.642407 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:33.664863 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:33.664834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:34.594722 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:34.594683 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-2pbn2" Jul 15 19:36:52.779344 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.779301 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f"] Jul 15 19:36:52.780382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.780115 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a8cd5e-744c-4041-bb96-b144aed53c59" containerName="registry-server" Jul 15 19:36:52.780382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.780141 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a8cd5e-744c-4041-bb96-b144aed53c59" containerName="registry-server" Jul 15 19:36:52.782139 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.780761 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a8cd5e-744c-4041-bb96-b144aed53c59" containerName="registry-server" Jul 15 19:36:52.785671 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.785652 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:36:52.787763 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.787744 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pptmj\"" Jul 15 19:36:52.793242 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.793223 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f"] Jul 15 19:36:52.864047 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.864020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28lx\" (UniqueName: \"kubernetes.io/projected/dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7-kube-api-access-c28lx\") pod \"limitador-operator-controller-manager-859d7b78bd-zd24f\" (UID: \"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:36:52.965196 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.965159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28lx\" (UniqueName: \"kubernetes.io/projected/dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7-kube-api-access-c28lx\") pod \"limitador-operator-controller-manager-859d7b78bd-zd24f\" (UID: \"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:36:52.987382 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:52.987346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28lx\" (UniqueName: \"kubernetes.io/projected/dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7-kube-api-access-c28lx\") pod \"limitador-operator-controller-manager-859d7b78bd-zd24f\" (UID: \"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:36:53.096359 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:53.096279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:36:53.227534 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:53.227507 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f"] Jul 15 19:36:53.229356 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:36:53.229316 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6fa976_9fa3_49f8_a9f8_00433d4a9bd7.slice/crio-f99e91db1c38935a41a85dcfbe8320e8830cf4603ccb46871d556f1bdd142ea2 WatchSource:0}: Error finding container f99e91db1c38935a41a85dcfbe8320e8830cf4603ccb46871d556f1bdd142ea2: Status 404 returned error can't find the container with id f99e91db1c38935a41a85dcfbe8320e8830cf4603ccb46871d556f1bdd142ea2 Jul 15 19:36:53.654312 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:53.654275 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" event={"ID":"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7","Type":"ContainerStarted","Data":"f99e91db1c38935a41a85dcfbe8320e8830cf4603ccb46871d556f1bdd142ea2"} Jul 15 19:36:55.664716 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:55.664677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" event={"ID":"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7","Type":"ContainerStarted","Data":"ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb"} Jul 15 19:36:55.665141 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:55.664834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:36:55.683754 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:55.683708 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" podStartSLOduration=1.8524319409999999 podStartE2EDuration="3.683695686s" totalImagesPullingTime="1.831263745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:36:55.682046146 +0000 UTC m=+497.543250500" watchObservedRunningTime="2026-07-15 19:36:55.683695686 +0000 UTC m=+497.544900038" Jul 15 19:36:59.505863 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.505832 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw"] Jul 15 19:36:59.511376 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.511357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.514052 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.514032 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hfv28\"" Jul 15 19:36:59.538579 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.538547 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw"] Jul 15 19:36:59.616625 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.616589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlh9\" (UniqueName: \"kubernetes.io/projected/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-kube-api-access-hdlh9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.616788 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.616678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.717913 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.717879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlh9\" (UniqueName: \"kubernetes.io/projected/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-kube-api-access-hdlh9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.718088 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.717948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.718280 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.718263 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.728588 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.728556 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlh9\" (UniqueName: \"kubernetes.io/projected/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-kube-api-access-hdlh9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.821218 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.821134 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:36:59.953875 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:36:59.953852 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw"] Jul 15 19:36:59.956479 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:36:59.956449 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5217f9ae_3e4f_4298_a318_cf6bc4d2aca4.slice/crio-1ffdee41da6c1da0a08a2a533aaad5f31e51e40b450d4d7b98b1f84f6a28154e WatchSource:0}: Error finding container 1ffdee41da6c1da0a08a2a533aaad5f31e51e40b450d4d7b98b1f84f6a28154e: Status 404 returned error can't find the container with id 1ffdee41da6c1da0a08a2a533aaad5f31e51e40b450d4d7b98b1f84f6a28154e Jul 15 19:37:00.695583 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:00.695546 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" event={"ID":"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4","Type":"ContainerStarted","Data":"1ffdee41da6c1da0a08a2a533aaad5f31e51e40b450d4d7b98b1f84f6a28154e"} Jul 15 19:37:03.565068 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.565035 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2"] Jul 15 19:37:03.568366 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.568341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" Jul 15 19:37:03.570815 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.570773 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 15 19:37:03.570949 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.570816 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4vrwk\"" Jul 15 19:37:03.575569 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.575548 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2"] Jul 15 19:37:03.652464 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.652436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8-kube-api-access-k48sz\") pod \"dns-operator-controller-manager-864dd78c4c-cnxc2\" (UID: \"b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" Jul 15 19:37:03.753117 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.753046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8-kube-api-access-k48sz\") pod \"dns-operator-controller-manager-864dd78c4c-cnxc2\" (UID: \"b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" Jul 15 19:37:03.764369 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.764320 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8-kube-api-access-k48sz\") pod \"dns-operator-controller-manager-864dd78c4c-cnxc2\" (UID: \"b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" Jul 15 19:37:03.880330 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:03.880248 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" Jul 15 19:37:04.025361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:04.025326 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2"] Jul 15 19:37:05.167913 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:37:05.167883 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dcacd9_cbb2_4a7d_b4be_ce77844b9aa8.slice/crio-694c4fd276bbbdefab7bc3000e7f90c445fa998c8263e92a5c4655860efb7060 WatchSource:0}: Error finding container 694c4fd276bbbdefab7bc3000e7f90c445fa998c8263e92a5c4655860efb7060: Status 404 returned error can't find the container with id 694c4fd276bbbdefab7bc3000e7f90c445fa998c8263e92a5c4655860efb7060 Jul 15 19:37:05.714040 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:05.714003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" event={"ID":"b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8","Type":"ContainerStarted","Data":"694c4fd276bbbdefab7bc3000e7f90c445fa998c8263e92a5c4655860efb7060"} Jul 15 19:37:05.715575 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:05.715546 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" event={"ID":"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4","Type":"ContainerStarted","Data":"0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2"} Jul 15 19:37:05.715693 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:05.715656 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:05.734033 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:05.733987 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" podStartSLOduration=1.468921781 podStartE2EDuration="6.733968458s" totalImagesPullingTime="5.265046677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:05.73277702 +0000 UTC m=+507.593981411" watchObservedRunningTime="2026-07-15 19:37:05.733968458 +0000 UTC m=+507.595172813" Jul 15 19:37:06.673656 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:06.673625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:07.727533 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.727491 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" event={"ID":"b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8","Type":"ContainerStarted","Data":"174fa23b26fd6480486c06b2a465d67d7e7ba0f685883082c6c987a73ffb2e6a"} Jul 15 19:37:07.728012 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.727611 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" Jul 15 19:37:07.743107 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.743060 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" podStartSLOduration=2.786665235 podStartE2EDuration="4.743046062s" totalImagesPullingTime="1.956380827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:07.740850095 +0000 UTC m=+509.602054461" watchObservedRunningTime="2026-07-15 19:37:07.743046062 +0000 UTC m=+509.604250480" Jul 15 19:37:07.954490 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.954454 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm"] Jul 15 19:37:07.957786 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.957767 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:07.959728 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.959707 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 19:37:07.960020 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.959998 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 19:37:07.960086 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.960031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dm48d\"" Jul 15 19:37:07.964649 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:07.964627 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm"] Jul 15 19:37:08.092119 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.092038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkps\" (UniqueName: \"kubernetes.io/projected/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-kube-api-access-kmkps\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.092119 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.092075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.092291 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.092159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.193290 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.193258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkps\" (UniqueName: \"kubernetes.io/projected/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-kube-api-access-kmkps\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.193290 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.193291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.193494 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.193328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.193877 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:08.193838 2573 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 15 19:37:08.193969 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:08.193953 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-plugin-serving-cert podName:61fd86f6-ba8a-4893-8773-315fe9d1f8d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:37:08.693908877 +0000 UTC m=+510.555113225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-fcwlm" (UID: "61fd86f6-ba8a-4893-8773-315fe9d1f8d6") : secret "plugin-serving-cert" not found Jul 15 19:37:08.194538 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.194505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.202460 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.202434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkps\" (UniqueName: \"kubernetes.io/projected/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-kube-api-access-kmkps\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.697191 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.697161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.699649 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.699619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/61fd86f6-ba8a-4893-8773-315fe9d1f8d6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-fcwlm\" (UID: \"61fd86f6-ba8a-4893-8773-315fe9d1f8d6\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.868086 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.868047 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" Jul 15 19:37:08.999412 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:08.999380 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm"] Jul 15 19:37:09.000913 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:37:09.000879 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61fd86f6_ba8a_4893_8773_315fe9d1f8d6.slice/crio-d9087af562cf8f986f300ed5055e028c8286f27602293acfaa479b2b6e11dcb2 WatchSource:0}: Error finding container d9087af562cf8f986f300ed5055e028c8286f27602293acfaa479b2b6e11dcb2: Status 404 returned error can't find the container with id d9087af562cf8f986f300ed5055e028c8286f27602293acfaa479b2b6e11dcb2 Jul 15 19:37:09.736848 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:09.736791 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" event={"ID":"61fd86f6-ba8a-4893-8773-315fe9d1f8d6","Type":"ContainerStarted","Data":"d9087af562cf8f986f300ed5055e028c8286f27602293acfaa479b2b6e11dcb2"} Jul 15 19:37:16.723589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:16.723556 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:17.813754 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.813712 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz"] Jul 15 19:37:17.835558 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.835475 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz"] Jul 15 19:37:17.835720 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.835584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:17.925705 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.925668 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz"] Jul 15 19:37:17.926019 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:17.925998 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-wd9k7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" podUID="2261937c-547a-41cf-9eed-fbec28cf8211" Jul 15 19:37:17.958261 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.958226 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l"] Jul 15 19:37:17.963381 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.963359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:17.972361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.972336 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l"] Jul 15 19:37:17.987272 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.987229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9k7\" (UniqueName: \"kubernetes.io/projected/2261937c-547a-41cf-9eed-fbec28cf8211-kube-api-access-wd9k7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:17.987397 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:17.987279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2261937c-547a-41cf-9eed-fbec28cf8211-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.088705 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.088615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2261937c-547a-41cf-9eed-fbec28cf8211-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.088895 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.088735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zms5g\" (UniqueName: \"kubernetes.io/projected/81c928f7-9672-48ab-b34e-d2745343ca01-kube-api-access-zms5g\") pod \"kuadrant-operator-controller-manager-55467d9f7-8258l\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:18.088895 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.088790 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81c928f7-9672-48ab-b34e-d2745343ca01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8258l\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:18.088895 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.088870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9k7\" (UniqueName: \"kubernetes.io/projected/2261937c-547a-41cf-9eed-fbec28cf8211-kube-api-access-wd9k7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.089156 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.089133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2261937c-547a-41cf-9eed-fbec28cf8211-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.100364 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.100334 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9k7\" (UniqueName: \"kubernetes.io/projected/2261937c-547a-41cf-9eed-fbec28cf8211-kube-api-access-wd9k7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.189609 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.189575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zms5g\" (UniqueName: \"kubernetes.io/projected/81c928f7-9672-48ab-b34e-d2745343ca01-kube-api-access-zms5g\") pod \"kuadrant-operator-controller-manager-55467d9f7-8258l\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:18.189789 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.189673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81c928f7-9672-48ab-b34e-d2745343ca01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8258l\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:18.190083 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.190058 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81c928f7-9672-48ab-b34e-d2745343ca01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8258l\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:18.201786 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.201762 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zms5g\" (UniqueName: \"kubernetes.io/projected/81c928f7-9672-48ab-b34e-d2745343ca01-kube-api-access-zms5g\") pod \"kuadrant-operator-controller-manager-55467d9f7-8258l\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:18.275753 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.275716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:18.406371 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.406337 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l"] Jul 15 19:37:18.421653 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.421583 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l"] Jul 15 19:37:18.426071 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.426041 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv"] Jul 15 19:37:18.433661 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.433629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.434369 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.434331 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz"] Jul 15 19:37:18.439987 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.439928 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz"] Jul 15 19:37:18.442055 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.442029 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv"] Jul 15 19:37:18.450289 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.450133 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw"] Jul 15 19:37:18.450416 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.450358 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" containerName="manager" containerID="cri-o://0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2" gracePeriod=2 Jul 15 19:37:18.460655 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.460625 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw"] Jul 15 19:37:18.471568 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.471537 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c"] Jul 15 19:37:18.472081 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.472053 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" containerName="manager" Jul 15 19:37:18.472081 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.472078 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" containerName="manager" Jul 15 19:37:18.472255 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.472173 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" containerName="manager" Jul 15 19:37:18.475528 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.475501 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f"] Jul 15 19:37:18.475648 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.475623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.475726 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.475703 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" containerName="manager" containerID="cri-o://ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb" gracePeriod=2 Jul 15 19:37:18.485354 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.485022 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f"] Jul 15 19:37:18.485354 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.485230 2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-zd24f\": pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:18.487554 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.487322 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:18.487554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.487529 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c"] Jul 15 19:37:18.488925 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.488886 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:18.494516 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.494393 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t"] Jul 15 19:37:18.494866 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.494847 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" containerName="manager" Jul 15 19:37:18.494866 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.494866 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" containerName="manager" Jul 15 19:37:18.495017 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.494985 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" containerName="manager" Jul 15 19:37:18.499231 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.499210 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" Jul 15 19:37:18.507748 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.507663 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t"] Jul 15 19:37:18.510383 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.510351 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:18.533669 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.533630 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:18.535020 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.534985 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:18.593249 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.593213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dd5bd8-ff41-43cf-b61b-3ceb34799013-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-hwdgv\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.593402 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.593273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgrv\" (UniqueName: \"kubernetes.io/projected/c6204216-aac1-4341-839c-84ed1a5497d2-kube-api-access-lqgrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5j75c\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.593402 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.593377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7b6z\" (UniqueName: \"kubernetes.io/projected/95dd5bd8-ff41-43cf-b61b-3ceb34799013-kube-api-access-j7b6z\") pod \"kuadrant-operator-controller-manager-55467d9f7-hwdgv\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.593488 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.593417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6204216-aac1-4341-839c-84ed1a5497d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5j75c\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.593488 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.593462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4gd\" (UniqueName: \"kubernetes.io/projected/403701c4-e563-4ed1-b4a3-0f10a99b4cf4-kube-api-access-7h4gd\") pod \"limitador-operator-controller-manager-859d7b78bd-4qv9t\" (UID: \"403701c4-e563-4ed1-b4a3-0f10a99b4cf4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" Jul 15 19:37:18.693944 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.693843 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:18.694155 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.694126 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4gd\" (UniqueName: \"kubernetes.io/projected/403701c4-e563-4ed1-b4a3-0f10a99b4cf4-kube-api-access-7h4gd\") pod \"limitador-operator-controller-manager-859d7b78bd-4qv9t\" (UID: \"403701c4-e563-4ed1-b4a3-0f10a99b4cf4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" Jul 15 19:37:18.694208 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.694188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dd5bd8-ff41-43cf-b61b-3ceb34799013-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-hwdgv\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.694261 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.694218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgrv\" (UniqueName: \"kubernetes.io/projected/c6204216-aac1-4341-839c-84ed1a5497d2-kube-api-access-lqgrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5j75c\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.694312 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.694273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7b6z\" (UniqueName: \"kubernetes.io/projected/95dd5bd8-ff41-43cf-b61b-3ceb34799013-kube-api-access-j7b6z\") pod \"kuadrant-operator-controller-manager-55467d9f7-hwdgv\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.694372 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.694315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6204216-aac1-4341-839c-84ed1a5497d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5j75c\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.694717 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.694669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6204216-aac1-4341-839c-84ed1a5497d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5j75c\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.694717 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.694694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dd5bd8-ff41-43cf-b61b-3ceb34799013-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-hwdgv\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.696656 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.696622 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:18.705495 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.705468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4gd\" (UniqueName: \"kubernetes.io/projected/403701c4-e563-4ed1-b4a3-0f10a99b4cf4-kube-api-access-7h4gd\") pod \"limitador-operator-controller-manager-859d7b78bd-4qv9t\" (UID: \"403701c4-e563-4ed1-b4a3-0f10a99b4cf4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" Jul 15 19:37:18.705620 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.705570 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7b6z\" (UniqueName: \"kubernetes.io/projected/95dd5bd8-ff41-43cf-b61b-3ceb34799013-kube-api-access-j7b6z\") pod \"kuadrant-operator-controller-manager-55467d9f7-hwdgv\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.705663 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.705638 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgrv\" (UniqueName: \"kubernetes.io/projected/c6204216-aac1-4341-839c-84ed1a5497d2-kube-api-access-lqgrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5j75c\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.733823 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.733771 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnxc2" Jul 15 19:37:18.751318 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.751291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:18.787349 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.787317 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.797563 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.797542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.800080 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:18.800034 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="2261937c-547a-41cf-9eed-fbec28cf8211" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:18.867448 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.867411 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:18.875021 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.874994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" Jul 15 19:37:18.895781 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.895757 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd9k7\" (UniqueName: \"kubernetes.io/projected/2261937c-547a-41cf-9eed-fbec28cf8211-kube-api-access-wd9k7\") pod \"2261937c-547a-41cf-9eed-fbec28cf8211\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " Jul 15 19:37:18.895944 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.895875 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2261937c-547a-41cf-9eed-fbec28cf8211-extensions-socket-volume\") pod \"2261937c-547a-41cf-9eed-fbec28cf8211\" (UID: \"2261937c-547a-41cf-9eed-fbec28cf8211\") " Jul 15 19:37:18.896196 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.896171 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2261937c-547a-41cf-9eed-fbec28cf8211-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2261937c-547a-41cf-9eed-fbec28cf8211" (UID: "2261937c-547a-41cf-9eed-fbec28cf8211"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:18.898312 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.898286 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2261937c-547a-41cf-9eed-fbec28cf8211-kube-api-access-wd9k7" (OuterVolumeSpecName: "kube-api-access-wd9k7") pod "2261937c-547a-41cf-9eed-fbec28cf8211" (UID: "2261937c-547a-41cf-9eed-fbec28cf8211"). InnerVolumeSpecName "kube-api-access-wd9k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:37:18.997284 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.997204 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2261937c-547a-41cf-9eed-fbec28cf8211-extensions-socket-volume\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:18.997284 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:18.997233 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd9k7\" (UniqueName: \"kubernetes.io/projected/2261937c-547a-41cf-9eed-fbec28cf8211-kube-api-access-wd9k7\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:19.793290 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:19.793257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:19.795867 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:19.795829 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="2261937c-547a-41cf-9eed-fbec28cf8211" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:19.802721 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:19.802685 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-bppvz\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="2261937c-547a-41cf-9eed-fbec28cf8211" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-bppvz" Jul 15 19:37:20.696994 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:20.696950 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2261937c-547a-41cf-9eed-fbec28cf8211" path="/var/lib/kubelet/pods/2261937c-547a-41cf-9eed-fbec28cf8211/volumes" Jul 15 19:37:30.604833 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.604425 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:30.604833 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.604776 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c"] Jul 15 19:37:30.606135 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:37:30.606106 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6204216_aac1_4341_839c_84ed1a5497d2.slice/crio-75d0481427280481ffc539132fe8684f7e8b5d9a952ab61ffb33ae2343c665f7 WatchSource:0}: Error finding container 75d0481427280481ffc539132fe8684f7e8b5d9a952ab61ffb33ae2343c665f7: Status 404 returned error can't find the container with id 75d0481427280481ffc539132fe8684f7e8b5d9a952ab61ffb33ae2343c665f7 Jul 15 19:37:30.607184 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.607146 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:30.608174 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.608135 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:30.610077 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.610049 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:30.612253 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.612219 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:30.634616 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.634590 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv"] Jul 15 19:37:30.645175 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:37:30.645147 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dd5bd8_ff41_43cf_b61b_3ceb34799013.slice/crio-9e2c122825389eef334209b755b2866c2f7a0d77aa1225b55fd0cc072cb61bf0 WatchSource:0}: Error finding container 9e2c122825389eef334209b755b2866c2f7a0d77aa1225b55fd0cc072cb61bf0: Status 404 returned error can't find the container with id 9e2c122825389eef334209b755b2866c2f7a0d77aa1225b55fd0cc072cb61bf0 Jul 15 19:37:30.645990 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.645967 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t"] Jul 15 19:37:30.647388 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:37:30.647309 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403701c4_e563_4ed1_b4a3_0f10a99b4cf4.slice/crio-f06cdbe03a1b2955dbdd064dc1e7527c13649e21587655bf429144bfa69ae382 WatchSource:0}: Error finding container f06cdbe03a1b2955dbdd064dc1e7527c13649e21587655bf429144bfa69ae382: Status 404 returned error can't find the container with id f06cdbe03a1b2955dbdd064dc1e7527c13649e21587655bf429144bfa69ae382 Jul 15 19:37:30.694900 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.694866 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-extensions-socket-volume\") pod \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " Jul 15 19:37:30.695064 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.694943 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c28lx\" (UniqueName: \"kubernetes.io/projected/dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7-kube-api-access-c28lx\") pod \"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7\" (UID: \"dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7\") " Jul 15 19:37:30.695064 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.694988 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdlh9\" (UniqueName: \"kubernetes.io/projected/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-kube-api-access-hdlh9\") pod \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\" (UID: \"5217f9ae-3e4f-4298-a318-cf6bc4d2aca4\") " Jul 15 19:37:30.695485 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.695363 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" (UID: "5217f9ae-3e4f-4298-a318-cf6bc4d2aca4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:30.697537 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.697507 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-kube-api-access-hdlh9" (OuterVolumeSpecName: "kube-api-access-hdlh9") pod "5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" (UID: "5217f9ae-3e4f-4298-a318-cf6bc4d2aca4"). InnerVolumeSpecName "kube-api-access-hdlh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:37:30.698269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.698241 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7-kube-api-access-c28lx" (OuterVolumeSpecName: "kube-api-access-c28lx") pod "dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" (UID: "dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7"). InnerVolumeSpecName "kube-api-access-c28lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:37:30.796330 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.796293 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-extensions-socket-volume\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:30.796330 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.796326 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28lx\" (UniqueName: \"kubernetes.io/projected/dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7-kube-api-access-c28lx\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:30.796330 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.796335 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdlh9\" (UniqueName: \"kubernetes.io/projected/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4-kube-api-access-hdlh9\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:30.849100 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.849059 2573 generic.go:356] "Generic (PLEG): container finished" podID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" containerID="0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2" exitCode=0 Jul 15 19:37:30.849545 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.849107 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:30.849545 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.849165 2573 scope.go:117] "RemoveContainer" containerID="0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2" Jul 15 19:37:30.850619 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.850593 2573 generic.go:356] "Generic (PLEG): container finished" podID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" containerID="ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb" exitCode=0 Jul 15 19:37:30.850741 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.850651 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:30.851950 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.851874 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:30.853111 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.853085 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" event={"ID":"61fd86f6-ba8a-4893-8773-315fe9d1f8d6","Type":"ContainerStarted","Data":"5ed37bb81bdd90f3c38aedabf99491d751340afb39da52068bf21ab2056a9035"} Jul 15 19:37:30.853793 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.853662 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:30.855290 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.854962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" event={"ID":"95dd5bd8-ff41-43cf-b61b-3ceb34799013","Type":"ContainerStarted","Data":"9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0"} Jul 15 19:37:30.855290 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.855012 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" event={"ID":"95dd5bd8-ff41-43cf-b61b-3ceb34799013","Type":"ContainerStarted","Data":"9e2c122825389eef334209b755b2866c2f7a0d77aa1225b55fd0cc072cb61bf0"} Jul 15 19:37:30.855290 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.855038 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:30.855642 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.855619 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:30.856922 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.856898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" event={"ID":"403701c4-e563-4ed1-b4a3-0f10a99b4cf4","Type":"ContainerStarted","Data":"37bcc77b0e00f80113183905b46b4a692a90cb90958286f3570270ec1d5cf44f"} Jul 15 19:37:30.857028 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.856946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" event={"ID":"403701c4-e563-4ed1-b4a3-0f10a99b4cf4","Type":"ContainerStarted","Data":"f06cdbe03a1b2955dbdd064dc1e7527c13649e21587655bf429144bfa69ae382"} Jul 15 19:37:30.857028 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.856991 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" Jul 15 19:37:30.857503 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.857482 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:30.858577 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.858554 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" event={"ID":"c6204216-aac1-4341-839c-84ed1a5497d2","Type":"ContainerStarted","Data":"2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb"} Jul 15 19:37:30.858678 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.858592 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" event={"ID":"c6204216-aac1-4341-839c-84ed1a5497d2","Type":"ContainerStarted","Data":"75d0481427280481ffc539132fe8684f7e8b5d9a952ab61ffb33ae2343c665f7"} Jul 15 19:37:30.858678 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.858622 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:30.860163 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.860140 2573 scope.go:117] "RemoveContainer" containerID="0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2" Jul 15 19:37:30.860522 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.860495 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2\": container with ID starting with 0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2 not found: ID does not exist" containerID="0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2" Jul 15 19:37:30.860601 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.860529 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2"} err="failed to get container status \"0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2\": rpc error: code = NotFound desc = could not find container \"0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2\": container with ID starting with 0863cf907959b1b1feddf43b95ef3f673ed2ec792ef495f2a2a0f1ee939ff2a2 not found: ID does not exist" Jul 15 19:37:30.860601 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.860548 2573 scope.go:117] "RemoveContainer" containerID="ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb" Jul 15 19:37:30.868520 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.868504 2573 scope.go:117] "RemoveContainer" containerID="ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb" Jul 15 19:37:30.868757 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.868738 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb\": container with ID starting with ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb not found: ID does not exist" containerID="ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb" Jul 15 19:37:30.868838 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.868767 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb"} err="failed to get container status \"ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb\": rpc error: code = NotFound desc = could not find container \"ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb\": container with ID starting with ce84474f2d75a0693cff381ef9c8f3ab211ba5359b7ee9e1b683541f9bd7cccb not found: ID does not exist" Jul 15 19:37:30.875780 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.875728 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-fcwlm" podStartSLOduration=2.357305147 podStartE2EDuration="23.875714594s" totalImagesPullingTime="21.518409447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:30.875064174 +0000 UTC m=+532.736268527" watchObservedRunningTime="2026-07-15 19:37:30.875714594 +0000 UTC m=+532.736918946" Jul 15 19:37:30.900649 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.899899 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" podStartSLOduration=12.899878034 podStartE2EDuration="12.899878034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:30.89812003 +0000 UTC m=+532.759324404" watchObservedRunningTime="2026-07-15 19:37:30.899878034 +0000 UTC m=+532.761082389" Jul 15 19:37:30.901066 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.900908 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:30.902642 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.902601 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zd24f\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zd24f" Jul 15 19:37:30.920607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.920555 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" podStartSLOduration=12.920535485 podStartE2EDuration="12.920535485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:30.91998841 +0000 UTC m=+532.781192763" watchObservedRunningTime="2026-07-15 19:37:30.920535485 +0000 UTC m=+532.781739837" Jul 15 19:37:30.939536 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:30.939485 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" podStartSLOduration=12.93947133 podStartE2EDuration="12.93947133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:30.937960722 +0000 UTC m=+532.799165097" watchObservedRunningTime="2026-07-15 19:37:30.93947133 +0000 UTC m=+532.800675683" Jul 15 19:37:30.939885 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:30.939859 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fbhbw\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fbhbw" Jul 15 19:37:32.695514 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:32.695475 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5217f9ae-3e4f-4298-a318-cf6bc4d2aca4" path="/var/lib/kubelet/pods/5217f9ae-3e4f-4298-a318-cf6bc4d2aca4/volumes" Jul 15 19:37:32.695904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:32.695845 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7" path="/var/lib/kubelet/pods/dc6fa976-9fa3-49f8-a9f8-00433d4a9bd7/volumes" Jul 15 19:37:32.958423 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:32.958380 2573 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-8258l_kuadrant-system_81c928f7-9672-48ab-b34e-d2745343ca01_0(1258b9ca0c39285a12f7eaf14591eaa7c7421187311addca601c08bda1115587): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-8258l to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 15 19:37:32.958488 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:32.958450 2573 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-8258l_kuadrant-system_81c928f7-9672-48ab-b34e-d2745343ca01_0(1258b9ca0c39285a12f7eaf14591eaa7c7421187311addca601c08bda1115587): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-8258l to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:33.873481 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:33.873449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:33.875401 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:33.875366 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8258l\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="81c928f7-9672-48ab-b34e-d2745343ca01" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:33.879785 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:33.879766 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:33.881870 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:33.881846 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8258l\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="81c928f7-9672-48ab-b34e-d2745343ca01" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:33.920589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:33.920560 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81c928f7-9672-48ab-b34e-d2745343ca01-extensions-socket-volume\") pod \"81c928f7-9672-48ab-b34e-d2745343ca01\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " Jul 15 19:37:33.920714 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:33.920604 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zms5g\" (UniqueName: \"kubernetes.io/projected/81c928f7-9672-48ab-b34e-d2745343ca01-kube-api-access-zms5g\") pod \"81c928f7-9672-48ab-b34e-d2745343ca01\" (UID: \"81c928f7-9672-48ab-b34e-d2745343ca01\") " Jul 15 19:37:33.920851 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:33.920829 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c928f7-9672-48ab-b34e-d2745343ca01-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "81c928f7-9672-48ab-b34e-d2745343ca01" (UID: "81c928f7-9672-48ab-b34e-d2745343ca01"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:33.922758 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:33.922726 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c928f7-9672-48ab-b34e-d2745343ca01-kube-api-access-zms5g" (OuterVolumeSpecName: "kube-api-access-zms5g") pod "81c928f7-9672-48ab-b34e-d2745343ca01" (UID: "81c928f7-9672-48ab-b34e-d2745343ca01"). InnerVolumeSpecName "kube-api-access-zms5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:37:34.021140 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:34.021108 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81c928f7-9672-48ab-b34e-d2745343ca01-extensions-socket-volume\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:34.021140 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:34.021134 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zms5g\" (UniqueName: \"kubernetes.io/projected/81c928f7-9672-48ab-b34e-d2745343ca01-kube-api-access-zms5g\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:34.695352 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:34.695321 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c928f7-9672-48ab-b34e-d2745343ca01" path="/var/lib/kubelet/pods/81c928f7-9672-48ab-b34e-d2745343ca01/volumes" Jul 15 19:37:34.877564 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:34.877537 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:34.879455 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:34.879425 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8258l\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="81c928f7-9672-48ab-b34e-d2745343ca01" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:34.881414 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:34.881387 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8258l\" is forbidden: User \"system:node:ip-10-0-142-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-40.ec2.internal' and this object" podUID="81c928f7-9672-48ab-b34e-d2745343ca01" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8258l" Jul 15 19:37:41.867523 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:41.867486 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:41.867962 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:41.867569 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:41.867962 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:41.867612 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4qv9t" Jul 15 19:37:41.933894 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:41.933864 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c"] Jul 15 19:37:41.934112 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:41.934079 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" podUID="c6204216-aac1-4341-839c-84ed1a5497d2" containerName="manager" containerID="cri-o://2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb" gracePeriod=10 Jul 15 19:37:42.197126 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.197096 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:42.279287 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.279254 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg"] Jul 15 19:37:42.279639 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.279625 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6204216-aac1-4341-839c-84ed1a5497d2" containerName="manager" Jul 15 19:37:42.279685 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.279641 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6204216-aac1-4341-839c-84ed1a5497d2" containerName="manager" Jul 15 19:37:42.279723 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.279706 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6204216-aac1-4341-839c-84ed1a5497d2" containerName="manager" Jul 15 19:37:42.281635 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.281619 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.291016 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.290995 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqgrv\" (UniqueName: \"kubernetes.io/projected/c6204216-aac1-4341-839c-84ed1a5497d2-kube-api-access-lqgrv\") pod \"c6204216-aac1-4341-839c-84ed1a5497d2\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " Jul 15 19:37:42.291125 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.291080 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6204216-aac1-4341-839c-84ed1a5497d2-extensions-socket-volume\") pod \"c6204216-aac1-4341-839c-84ed1a5497d2\" (UID: \"c6204216-aac1-4341-839c-84ed1a5497d2\") " Jul 15 19:37:42.291430 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.291407 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6204216-aac1-4341-839c-84ed1a5497d2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c6204216-aac1-4341-839c-84ed1a5497d2" (UID: "c6204216-aac1-4341-839c-84ed1a5497d2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:42.293466 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.293436 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6204216-aac1-4341-839c-84ed1a5497d2-kube-api-access-lqgrv" (OuterVolumeSpecName: "kube-api-access-lqgrv") pod "c6204216-aac1-4341-839c-84ed1a5497d2" (UID: "c6204216-aac1-4341-839c-84ed1a5497d2"). InnerVolumeSpecName "kube-api-access-lqgrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:37:42.296011 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.295987 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg"] Jul 15 19:37:42.391986 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.391942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e04e4309-6708-42b0-b61b-f71b7037a26f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wg5bg\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.392184 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.391998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kswhr\" (UniqueName: \"kubernetes.io/projected/e04e4309-6708-42b0-b61b-f71b7037a26f-kube-api-access-kswhr\") pod \"kuadrant-operator-controller-manager-86878c769f-wg5bg\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.392184 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.392104 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6204216-aac1-4341-839c-84ed1a5497d2-extensions-socket-volume\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:42.392184 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.392128 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgrv\" (UniqueName: \"kubernetes.io/projected/c6204216-aac1-4341-839c-84ed1a5497d2-kube-api-access-lqgrv\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:42.492791 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.492701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e04e4309-6708-42b0-b61b-f71b7037a26f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wg5bg\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.492791 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.492748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswhr\" (UniqueName: \"kubernetes.io/projected/e04e4309-6708-42b0-b61b-f71b7037a26f-kube-api-access-kswhr\") pod \"kuadrant-operator-controller-manager-86878c769f-wg5bg\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.493122 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.493100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e04e4309-6708-42b0-b61b-f71b7037a26f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wg5bg\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.506853 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.506830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kswhr\" (UniqueName: \"kubernetes.io/projected/e04e4309-6708-42b0-b61b-f71b7037a26f-kube-api-access-kswhr\") pod \"kuadrant-operator-controller-manager-86878c769f-wg5bg\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.591764 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.591720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.725530 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.725509 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg"] Jul 15 19:37:42.727636 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:37:42.727605 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e4309_6708_42b0_b61b_f71b7037a26f.slice/crio-efa91f4e821d6e333194349c64ec3bb3e0d281096e38d538a4f259eb00d899aa WatchSource:0}: Error finding container efa91f4e821d6e333194349c64ec3bb3e0d281096e38d538a4f259eb00d899aa: Status 404 returned error can't find the container with id efa91f4e821d6e333194349c64ec3bb3e0d281096e38d538a4f259eb00d899aa Jul 15 19:37:42.918085 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.918048 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" event={"ID":"e04e4309-6708-42b0-b61b-f71b7037a26f","Type":"ContainerStarted","Data":"57c32f0bb304950b65a4cc9a60d66147ca340d8fe600bf7b375fee168897af32"} Jul 15 19:37:42.918505 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.918149 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:42.918505 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.918169 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" event={"ID":"e04e4309-6708-42b0-b61b-f71b7037a26f","Type":"ContainerStarted","Data":"efa91f4e821d6e333194349c64ec3bb3e0d281096e38d538a4f259eb00d899aa"} Jul 15 19:37:42.919393 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.919370 2573 generic.go:356] "Generic (PLEG): container finished" podID="c6204216-aac1-4341-839c-84ed1a5497d2" containerID="2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb" exitCode=0 Jul 15 19:37:42.919498 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.919413 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" event={"ID":"c6204216-aac1-4341-839c-84ed1a5497d2","Type":"ContainerDied","Data":"2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb"} Jul 15 19:37:42.919498 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.919429 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" Jul 15 19:37:42.919498 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.919448 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c" event={"ID":"c6204216-aac1-4341-839c-84ed1a5497d2","Type":"ContainerDied","Data":"75d0481427280481ffc539132fe8684f7e8b5d9a952ab61ffb33ae2343c665f7"} Jul 15 19:37:42.919498 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.919469 2573 scope.go:117] "RemoveContainer" containerID="2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb" Jul 15 19:37:42.927901 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.927865 2573 scope.go:117] "RemoveContainer" containerID="2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb" Jul 15 19:37:42.928169 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:42.928142 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb\": container with ID starting with 2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb not found: ID does not exist" containerID="2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb" Jul 15 19:37:42.928261 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.928182 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb"} err="failed to get container status \"2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb\": rpc error: code = NotFound desc = could not find container \"2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb\": container with ID starting with 2079759d0b657dd0232fc0d48f29e5fc127489a85eba619a33c994493e04d4bb not found: ID does not exist" Jul 15 19:37:42.935697 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.935656 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" podStartSLOduration=0.935642641 podStartE2EDuration="935.642641ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:42.933997853 +0000 UTC m=+544.795202226" watchObservedRunningTime="2026-07-15 19:37:42.935642641 +0000 UTC m=+544.796846993" Jul 15 19:37:42.948288 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.948268 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c"] Jul 15 19:37:42.952749 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:42.952727 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5j75c"] Jul 15 19:37:44.695141 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:44.695109 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6204216-aac1-4341-839c-84ed1a5497d2" path="/var/lib/kubelet/pods/c6204216-aac1-4341-839c-84ed1a5497d2/volumes" Jul 15 19:37:53.927418 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:53.927389 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:37:53.976235 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:53.976201 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv"] Jul 15 19:37:53.976429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:53.976399 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" podUID="95dd5bd8-ff41-43cf-b61b-3ceb34799013" containerName="manager" containerID="cri-o://9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0" gracePeriod=10 Jul 15 19:37:54.234360 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.234335 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:54.294099 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.294064 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7b6z\" (UniqueName: \"kubernetes.io/projected/95dd5bd8-ff41-43cf-b61b-3ceb34799013-kube-api-access-j7b6z\") pod \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " Jul 15 19:37:54.294262 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.294106 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dd5bd8-ff41-43cf-b61b-3ceb34799013-extensions-socket-volume\") pod \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\" (UID: \"95dd5bd8-ff41-43cf-b61b-3ceb34799013\") " Jul 15 19:37:54.294498 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.294473 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95dd5bd8-ff41-43cf-b61b-3ceb34799013-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "95dd5bd8-ff41-43cf-b61b-3ceb34799013" (UID: "95dd5bd8-ff41-43cf-b61b-3ceb34799013"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:37:54.296430 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.296397 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95dd5bd8-ff41-43cf-b61b-3ceb34799013-kube-api-access-j7b6z" (OuterVolumeSpecName: "kube-api-access-j7b6z") pod "95dd5bd8-ff41-43cf-b61b-3ceb34799013" (UID: "95dd5bd8-ff41-43cf-b61b-3ceb34799013"). InnerVolumeSpecName "kube-api-access-j7b6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:37:54.395023 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.394993 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7b6z\" (UniqueName: \"kubernetes.io/projected/95dd5bd8-ff41-43cf-b61b-3ceb34799013-kube-api-access-j7b6z\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:54.395023 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.395022 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dd5bd8-ff41-43cf-b61b-3ceb34799013-extensions-socket-volume\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:37:54.978959 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.978872 2573 generic.go:356] "Generic (PLEG): container finished" podID="95dd5bd8-ff41-43cf-b61b-3ceb34799013" containerID="9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0" exitCode=0 Jul 15 19:37:54.978959 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.978949 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" Jul 15 19:37:54.979417 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.978970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" event={"ID":"95dd5bd8-ff41-43cf-b61b-3ceb34799013","Type":"ContainerDied","Data":"9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0"} Jul 15 19:37:54.979417 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.979079 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv" event={"ID":"95dd5bd8-ff41-43cf-b61b-3ceb34799013","Type":"ContainerDied","Data":"9e2c122825389eef334209b755b2866c2f7a0d77aa1225b55fd0cc072cb61bf0"} Jul 15 19:37:54.979417 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.979113 2573 scope.go:117] "RemoveContainer" containerID="9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0" Jul 15 19:37:54.987877 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.987860 2573 scope.go:117] "RemoveContainer" containerID="9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0" Jul 15 19:37:54.988166 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:37:54.988147 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0\": container with ID starting with 9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0 not found: ID does not exist" containerID="9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0" Jul 15 19:37:54.988230 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:54.988175 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0"} err="failed to get container status \"9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0\": rpc error: code = NotFound desc = could not find container \"9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0\": container with ID starting with 9f5e21fe0d2b2884c5f755f44dcddbf6a8839420624c2754c1bf2e0d327680a0 not found: ID does not exist" Jul 15 19:37:55.004018 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:55.003982 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv"] Jul 15 19:37:55.012483 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:55.012455 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hwdgv"] Jul 15 19:37:56.695420 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:56.695386 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95dd5bd8-ff41-43cf-b61b-3ceb34799013" path="/var/lib/kubelet/pods/95dd5bd8-ff41-43cf-b61b-3ceb34799013/volumes" Jul 15 19:37:58.311336 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.311306 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb"] Jul 15 19:37:58.311727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.311647 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95dd5bd8-ff41-43cf-b61b-3ceb34799013" containerName="manager" Jul 15 19:37:58.311727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.311659 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95dd5bd8-ff41-43cf-b61b-3ceb34799013" containerName="manager" Jul 15 19:37:58.311727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.311722 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="95dd5bd8-ff41-43cf-b61b-3ceb34799013" containerName="manager" Jul 15 19:37:58.366202 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.366166 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb"] Jul 15 19:37:58.366370 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.366295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.368373 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.368353 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gr545\"" Jul 15 19:37:58.431522 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431705 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431705 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431705 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431705 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz67w\" (UniqueName: \"kubernetes.io/projected/2b4cc9d4-1697-4718-9841-56e6fd40bde8-kube-api-access-fz67w\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.431904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.431842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.532856 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.532798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.532856 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.532859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.532879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.532922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.532947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.532978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.533010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.533053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz67w\" (UniqueName: \"kubernetes.io/projected/2b4cc9d4-1697-4718-9841-56e6fd40bde8-kube-api-access-fz67w\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.533086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533445 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.533290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533509 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.533478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.533567 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.533536 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.534000 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.533976 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.534159 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.534138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.535629 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.535601 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.535824 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.535784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.540748 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.540723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b4cc9d4-1697-4718-9841-56e6fd40bde8-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.541032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.541015 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz67w\" (UniqueName: \"kubernetes.io/projected/2b4cc9d4-1697-4718-9841-56e6fd40bde8-kube-api-access-fz67w\") pod \"maas-default-gateway-openshift-default-5896d4dd49-vc7zb\" (UID: \"2b4cc9d4-1697-4718-9841-56e6fd40bde8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.676982 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.676946 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:58.869350 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.869326 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb"] Jul 15 19:37:58.871428 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:37:58.871402 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4cc9d4_1697_4718_9841_56e6fd40bde8.slice/crio-fb8eb620d8e6a63e729faf279c5c4b17189f73e75ec266c8f1eef64c9a533675 WatchSource:0}: Error finding container fb8eb620d8e6a63e729faf279c5c4b17189f73e75ec266c8f1eef64c9a533675: Status 404 returned error can't find the container with id fb8eb620d8e6a63e729faf279c5c4b17189f73e75ec266c8f1eef64c9a533675 Jul 15 19:37:58.873492 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.873461 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:37:58.873563 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.873529 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:37:58.873563 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:58.873559 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:37:59.000860 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:59.000828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" event={"ID":"2b4cc9d4-1697-4718-9841-56e6fd40bde8","Type":"ContainerStarted","Data":"57ffb1f2e4b4ca112ef7c1ddbe3061baa23940f08f2c4b9df316cb19e96bf5f8"} Jul 15 19:37:59.001023 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:59.000878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" event={"ID":"2b4cc9d4-1697-4718-9841-56e6fd40bde8","Type":"ContainerStarted","Data":"fb8eb620d8e6a63e729faf279c5c4b17189f73e75ec266c8f1eef64c9a533675"} Jul 15 19:37:59.021630 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:59.021501 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" podStartSLOduration=1.02148347 podStartE2EDuration="1.02148347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:37:59.020685328 +0000 UTC m=+560.881889698" watchObservedRunningTime="2026-07-15 19:37:59.02148347 +0000 UTC m=+560.882687826" Jul 15 19:37:59.677340 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:59.677298 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:37:59.678821 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:59.678781 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vc7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:37:59.678938 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:37:59.678867 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" podUID="2b4cc9d4-1697-4718-9841-56e6fd40bde8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:38:00.681343 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:00.681312 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:38:01.009519 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:01.009433 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:38:01.010626 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:01.010599 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:38:13.861992 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:13.861959 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:38:13.864593 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:13.864571 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:13.866908 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:13.866885 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 19:38:13.877334 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:13.877311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:38:13.902100 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:13.902070 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:38:13.962566 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:13.962530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df252a02-9807-424b-b688-214d8671578f-config-file\") pod \"limitador-limitador-54d6d988d5-f84zs\" (UID: \"df252a02-9807-424b-b688-214d8671578f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:13.962750 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:13.962574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcfp\" (UniqueName: \"kubernetes.io/projected/df252a02-9807-424b-b688-214d8671578f-kube-api-access-kbcfp\") pod \"limitador-limitador-54d6d988d5-f84zs\" (UID: \"df252a02-9807-424b-b688-214d8671578f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:14.063316 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:14.063277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df252a02-9807-424b-b688-214d8671578f-config-file\") pod \"limitador-limitador-54d6d988d5-f84zs\" (UID: \"df252a02-9807-424b-b688-214d8671578f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:14.063492 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:14.063329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcfp\" (UniqueName: \"kubernetes.io/projected/df252a02-9807-424b-b688-214d8671578f-kube-api-access-kbcfp\") pod \"limitador-limitador-54d6d988d5-f84zs\" (UID: \"df252a02-9807-424b-b688-214d8671578f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:14.063977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:14.063958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df252a02-9807-424b-b688-214d8671578f-config-file\") pod \"limitador-limitador-54d6d988d5-f84zs\" (UID: \"df252a02-9807-424b-b688-214d8671578f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:14.070925 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:14.070902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcfp\" (UniqueName: \"kubernetes.io/projected/df252a02-9807-424b-b688-214d8671578f-kube-api-access-kbcfp\") pod \"limitador-limitador-54d6d988d5-f84zs\" (UID: \"df252a02-9807-424b-b688-214d8671578f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:14.174313 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:14.174280 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:14.308577 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:14.308553 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:38:14.311038 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:38:14.311006 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf252a02_9807_424b_b688_214d8671578f.slice/crio-b5b54f95a2034d9568589134f54fe13036a7df421e1afa52d7464844917b25a4 WatchSource:0}: Error finding container b5b54f95a2034d9568589134f54fe13036a7df421e1afa52d7464844917b25a4: Status 404 returned error can't find the container with id b5b54f95a2034d9568589134f54fe13036a7df421e1afa52d7464844917b25a4 Jul 15 19:38:15.078502 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:15.078463 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" event={"ID":"df252a02-9807-424b-b688-214d8671578f","Type":"ContainerStarted","Data":"b5b54f95a2034d9568589134f54fe13036a7df421e1afa52d7464844917b25a4"} Jul 15 19:38:18.096942 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:18.096902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" event={"ID":"df252a02-9807-424b-b688-214d8671578f","Type":"ContainerStarted","Data":"cc960627feeef9df59842168c63cbebdb17528a5cf7aa4b225b17858d1b5ae1e"} Jul 15 19:38:18.097451 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:18.097034 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:18.112030 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:18.111986 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" podStartSLOduration=2.361571194 podStartE2EDuration="5.111970515s" totalImagesPullingTime="2.750399321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:38:18.111483469 +0000 UTC m=+579.972687835" watchObservedRunningTime="2026-07-15 19:38:18.111970515 +0000 UTC m=+579.973175093" Jul 15 19:38:29.103332 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:29.103290 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-f84zs" Jul 15 19:38:38.588252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:38.588227 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:38:38.588980 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:38.588961 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:38:38.595959 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:38.595939 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:38:38.596432 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:38:38.596406 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:39:28.789096 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.789067 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-j8smn"] Jul 15 19:39:28.792198 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.792181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:28.794315 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.794282 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lvfsl\"" Jul 15 19:39:28.794417 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.794316 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 15 19:39:28.800405 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.800382 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-j8smn"] Jul 15 19:39:28.807041 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.807012 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:39:28.900346 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.900310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblhc\" (UniqueName: \"kubernetes.io/projected/814885eb-a332-4a7c-a701-7950cdc3e6ba-kube-api-access-sblhc\") pod \"payload-processing-f9b8d9b9b-j8smn\" (UID: \"814885eb-a332-4a7c-a701-7950cdc3e6ba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:28.900346 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:28.900350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814885eb-a332-4a7c-a701-7950cdc3e6ba-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-j8smn\" (UID: \"814885eb-a332-4a7c-a701-7950cdc3e6ba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:29.001261 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:29.001221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblhc\" (UniqueName: \"kubernetes.io/projected/814885eb-a332-4a7c-a701-7950cdc3e6ba-kube-api-access-sblhc\") pod \"payload-processing-f9b8d9b9b-j8smn\" (UID: \"814885eb-a332-4a7c-a701-7950cdc3e6ba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:29.001416 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:29.001263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814885eb-a332-4a7c-a701-7950cdc3e6ba-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-j8smn\" (UID: \"814885eb-a332-4a7c-a701-7950cdc3e6ba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:29.002040 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:29.002021 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814885eb-a332-4a7c-a701-7950cdc3e6ba-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-j8smn\" (UID: \"814885eb-a332-4a7c-a701-7950cdc3e6ba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:29.009410 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:29.009379 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblhc\" (UniqueName: \"kubernetes.io/projected/814885eb-a332-4a7c-a701-7950cdc3e6ba-kube-api-access-sblhc\") pod \"payload-processing-f9b8d9b9b-j8smn\" (UID: \"814885eb-a332-4a7c-a701-7950cdc3e6ba\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:29.102657 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:29.102575 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:29.446744 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:29.446712 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-j8smn"] Jul 15 19:39:29.447760 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:39:29.447725 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814885eb_a332_4a7c_a701_7950cdc3e6ba.slice/crio-23f10ab8226cc2526cf85b65fbfba82d5b01108bfc6504cbbf60837e91edeb69 WatchSource:0}: Error finding container 23f10ab8226cc2526cf85b65fbfba82d5b01108bfc6504cbbf60837e91edeb69: Status 404 returned error can't find the container with id 23f10ab8226cc2526cf85b65fbfba82d5b01108bfc6504cbbf60837e91edeb69 Jul 15 19:39:29.449753 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:29.449735 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:39:30.431364 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:30.431328 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" event={"ID":"814885eb-a332-4a7c-a701-7950cdc3e6ba","Type":"ContainerStarted","Data":"23f10ab8226cc2526cf85b65fbfba82d5b01108bfc6504cbbf60837e91edeb69"} Jul 15 19:39:31.439279 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:31.439245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" event={"ID":"814885eb-a332-4a7c-a701-7950cdc3e6ba","Type":"ContainerStarted","Data":"e5bb31f0d41fd514ed2403a7f99321d6c4aae47c78162b84a06e64c496a55149"} Jul 15 19:39:31.439691 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:31.439328 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:31.453666 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:31.453624 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" podStartSLOduration=1.9956964959999999 podStartE2EDuration="3.453610359s" totalImagesPullingTime="1.457913863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:39:31.453201554 +0000 UTC m=+653.314405907" watchObservedRunningTime="2026-07-15 19:39:31.453610359 +0000 UTC m=+653.314814711" Jul 15 19:39:37.472396 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:37.472361 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b4cc9d4-1697-4718-9841-56e6fd40bde8" containerID="57ffb1f2e4b4ca112ef7c1ddbe3061baa23940f08f2c4b9df316cb19e96bf5f8" exitCode=0 Jul 15 19:39:37.472396 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:37.472397 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" event={"ID":"2b4cc9d4-1697-4718-9841-56e6fd40bde8","Type":"ContainerDied","Data":"57ffb1f2e4b4ca112ef7c1ddbe3061baa23940f08f2c4b9df316cb19e96bf5f8"} Jul 15 19:39:37.472910 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:37.472724 2573 scope.go:117] "RemoveContainer" containerID="57ffb1f2e4b4ca112ef7c1ddbe3061baa23940f08f2c4b9df316cb19e96bf5f8" Jul 15 19:39:37.473696 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:37.473662 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:39:37.473798 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:37.473749 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:39:37.473896 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:37.473822 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:39:38.480196 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:38.480161 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" event={"ID":"2b4cc9d4-1697-4718-9841-56e6fd40bde8","Type":"ContainerStarted","Data":"1033c3acd0c96023951d902f3077eb98df64d2506de3f995d015aea6230b87a6"} Jul 15 19:39:38.685153 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:38.685114 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:39:38.685470 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:38.685302 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vc7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:39:38.685642 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:38.685619 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" podUID="2b4cc9d4-1697-4718-9841-56e6fd40bde8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:39:39.677725 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:39.677685 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vc7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:39:39.678375 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:39.678343 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" podUID="2b4cc9d4-1697-4718-9841-56e6fd40bde8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:39:40.677519 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:40.677474 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vc7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:39:40.677728 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:40.677551 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" podUID="2b4cc9d4-1697-4718-9841-56e6fd40bde8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:39:41.678165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:41.678122 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-vc7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 19:39:41.678695 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:41.678198 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" podUID="2b4cc9d4-1697-4718-9841-56e6fd40bde8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 19:39:42.446600 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:42.446568 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-j8smn" Jul 15 19:39:42.685136 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:42.685108 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:39:42.685594 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:42.685374 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:39:42.686196 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:42.686177 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-vc7zb" Jul 15 19:39:57.137049 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:39:57.137015 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:40:35.036063 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:35.036031 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:40:40.730868 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.730823 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5"] Jul 15 19:40:40.735662 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.735642 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.737844 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.737774 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-728wb\"" Jul 15 19:40:40.737977 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.737890 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 19:40:40.738041 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.737994 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 19:40:40.738686 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.738662 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 19:40:40.743001 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.742977 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5"] Jul 15 19:40:40.843607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.843575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.843786 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.843618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.843786 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.843649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.843786 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.843754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44pl\" (UniqueName: \"kubernetes.io/projected/06c84d29-fb69-4072-a137-82a2b3dd493d-kube-api-access-d44pl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.843936 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.843834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c84d29-fb69-4072-a137-82a2b3dd493d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.843936 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.843865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.944542 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.944509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44pl\" (UniqueName: \"kubernetes.io/projected/06c84d29-fb69-4072-a137-82a2b3dd493d-kube-api-access-d44pl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.944727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.944558 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c84d29-fb69-4072-a137-82a2b3dd493d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.944727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.944589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.944727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.944616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.944727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.944644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.944727 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.944672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.945101 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.945081 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.945175 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.945138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.945256 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.945235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.946998 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.946979 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c84d29-fb69-4072-a137-82a2b3dd493d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.947166 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.947149 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c84d29-fb69-4072-a137-82a2b3dd493d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:40.951711 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:40.951688 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44pl\" (UniqueName: \"kubernetes.io/projected/06c84d29-fb69-4072-a137-82a2b3dd493d-kube-api-access-d44pl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5sgr5\" (UID: \"06c84d29-fb69-4072-a137-82a2b3dd493d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:41.049361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:41.049275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:41.190695 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:41.190667 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5"] Jul 15 19:40:41.193259 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:40:41.193230 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c84d29_fb69_4072_a137_82a2b3dd493d.slice/crio-d5b23fb0f6cbeb9b4c6443ae624a9cf9bc1301bf3adfe3a031b5c0c22324e708 WatchSource:0}: Error finding container d5b23fb0f6cbeb9b4c6443ae624a9cf9bc1301bf3adfe3a031b5c0c22324e708: Status 404 returned error can't find the container with id d5b23fb0f6cbeb9b4c6443ae624a9cf9bc1301bf3adfe3a031b5c0c22324e708 Jul 15 19:40:41.800276 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:41.800233 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" event={"ID":"06c84d29-fb69-4072-a137-82a2b3dd493d","Type":"ContainerStarted","Data":"d5b23fb0f6cbeb9b4c6443ae624a9cf9bc1301bf3adfe3a031b5c0c22324e708"} Jul 15 19:40:42.367117 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:42.367078 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:40:42.806360 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:42.806320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" event={"ID":"06c84d29-fb69-4072-a137-82a2b3dd493d","Type":"ContainerStarted","Data":"bb7465716f32767ee8d5e3bf0019449e3078bd1569e8c93542e7196ad171ebc7"} Jul 15 19:40:42.825349 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:42.825301 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" podStartSLOduration=1.757396846 podStartE2EDuration="2.825287676s" totalImagesPullingTime="1.06789083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:40:42.824236945 +0000 UTC m=+724.685441316" watchObservedRunningTime="2026-07-15 19:40:42.825287676 +0000 UTC m=+724.686492030" Jul 15 19:40:51.050419 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:51.050382 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:51.050904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:51.050659 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:51.063265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:51.063235 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:40:51.861290 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:40:51.861255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5sgr5" Jul 15 19:41:18.130086 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:18.130051 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:41:24.435364 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:24.435324 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:41:28.338552 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.338521 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt"] Jul 15 19:41:28.343496 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.343474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.346715 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.346690 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 19:41:28.353514 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.353489 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt"] Jul 15 19:41:28.441909 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.441877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.442107 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.441924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knhm\" (UniqueName: \"kubernetes.io/projected/dcc6aaef-d7af-42f0-8410-0a65c03750c3-kube-api-access-6knhm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.442107 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.441965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.442107 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.442085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.442216 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.442137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.442216 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.442181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc6aaef-d7af-42f0-8410-0a65c03750c3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.542728 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.542697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.542957 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.542758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.542957 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.542786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.542957 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.542835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc6aaef-d7af-42f0-8410-0a65c03750c3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.542957 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.542871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.542957 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.542897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knhm\" (UniqueName: \"kubernetes.io/projected/dcc6aaef-d7af-42f0-8410-0a65c03750c3-kube-api-access-6knhm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.543242 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.543209 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.543372 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.543236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.543372 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.543339 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.545261 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.545244 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc6aaef-d7af-42f0-8410-0a65c03750c3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.545558 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.545541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc6aaef-d7af-42f0-8410-0a65c03750c3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.550033 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.550008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knhm\" (UniqueName: \"kubernetes.io/projected/dcc6aaef-d7af-42f0-8410-0a65c03750c3-kube-api-access-6knhm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt\" (UID: \"dcc6aaef-d7af-42f0-8410-0a65c03750c3\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.652837 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.652724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:28.791470 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:28.791446 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt"] Jul 15 19:41:28.793565 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:41:28.793526 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc6aaef_d7af_42f0_8410_0a65c03750c3.slice/crio-7c0334b7be1b7d00712dbd6da7f1a1337cb7c205816faddd9dacc88691614a5f WatchSource:0}: Error finding container 7c0334b7be1b7d00712dbd6da7f1a1337cb7c205816faddd9dacc88691614a5f: Status 404 returned error can't find the container with id 7c0334b7be1b7d00712dbd6da7f1a1337cb7c205816faddd9dacc88691614a5f Jul 15 19:41:29.027069 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:29.027038 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" event={"ID":"dcc6aaef-d7af-42f0-8410-0a65c03750c3","Type":"ContainerStarted","Data":"7c0334b7be1b7d00712dbd6da7f1a1337cb7c205816faddd9dacc88691614a5f"} Jul 15 19:41:30.033289 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:30.033250 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" event={"ID":"dcc6aaef-d7af-42f0-8410-0a65c03750c3","Type":"ContainerStarted","Data":"429a9132f5397ee70efc9ed97204dc050f448e9a9d079e5b736419875674d055"} Jul 15 19:41:30.048380 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:30.048327 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" podStartSLOduration=1.811313197 podStartE2EDuration="2.04830837s" totalImagesPullingTime="236.995173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:41:30.047503357 +0000 UTC m=+771.908707709" watchObservedRunningTime="2026-07-15 19:41:30.04830837 +0000 UTC m=+771.909512719" Jul 15 19:41:35.042906 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:35.042866 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:41:38.652874 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:38.652834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:38.653222 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:38.652885 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:38.665476 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:38.665449 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:41:39.088320 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:41:39.088291 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt" Jul 15 19:42:57.139304 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:42:57.139266 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:43:09.436695 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:09.436664 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:43:18.831643 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:18.831613 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:43:26.929106 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:26.929060 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:43:37.835468 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:37.835433 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:43:38.628192 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:38.628167 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:43:38.628192 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:38.628185 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:43:38.636069 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:38.636050 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:43:38.636328 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:43:38.636312 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:44:04.734289 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:44:04.734250 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:44:17.630950 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:44:17.630914 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:44:51.433856 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:44:51.433798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:45:02.934207 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:45:02.934116 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:45:11.640503 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:45:11.640469 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:45:20.831223 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:45:20.831185 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:45:31.941160 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:45:31.941124 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:45:43.233828 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:45:43.233766 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:45:54.636850 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:45:54.636799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:46:04.437961 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:46:04.437928 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:46:30.033846 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:46:30.033752 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:46:42.233201 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:46:42.233168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:47:04.931360 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:47:04.931326 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:47:16.741085 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:47:16.741045 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:47:32.628710 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:47:32.628675 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:47:41.830367 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:47:41.830336 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:47:50.228617 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:47:50.228580 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:48:02.834260 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:02.834170 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:48:11.637874 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:11.637838 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:48:38.476305 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:38.476265 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg"] Jul 15 19:48:38.476822 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:38.476504 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" podUID="e04e4309-6708-42b0-b61b-f71b7037a26f" containerName="manager" containerID="cri-o://57c32f0bb304950b65a4cc9a60d66147ca340d8fe600bf7b375fee168897af32" gracePeriod=10 Jul 15 19:48:38.657157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:38.657128 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:48:38.658118 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:38.658097 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:48:38.665071 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:38.665052 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:48:38.665614 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:38.665598 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:48:41.122446 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.122412 2573 generic.go:356] "Generic (PLEG): container finished" podID="e04e4309-6708-42b0-b61b-f71b7037a26f" containerID="57c32f0bb304950b65a4cc9a60d66147ca340d8fe600bf7b375fee168897af32" exitCode=0 Jul 15 19:48:41.122829 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.122485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" event={"ID":"e04e4309-6708-42b0-b61b-f71b7037a26f","Type":"ContainerDied","Data":"57c32f0bb304950b65a4cc9a60d66147ca340d8fe600bf7b375fee168897af32"} Jul 15 19:48:41.535635 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.535607 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:48:41.626877 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.626844 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kswhr\" (UniqueName: \"kubernetes.io/projected/e04e4309-6708-42b0-b61b-f71b7037a26f-kube-api-access-kswhr\") pod \"e04e4309-6708-42b0-b61b-f71b7037a26f\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " Jul 15 19:48:41.627034 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.626949 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e04e4309-6708-42b0-b61b-f71b7037a26f-extensions-socket-volume\") pod \"e04e4309-6708-42b0-b61b-f71b7037a26f\" (UID: \"e04e4309-6708-42b0-b61b-f71b7037a26f\") " Jul 15 19:48:41.627338 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.627312 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04e4309-6708-42b0-b61b-f71b7037a26f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e04e4309-6708-42b0-b61b-f71b7037a26f" (UID: "e04e4309-6708-42b0-b61b-f71b7037a26f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:48:41.628996 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.628978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04e4309-6708-42b0-b61b-f71b7037a26f-kube-api-access-kswhr" (OuterVolumeSpecName: "kube-api-access-kswhr") pod "e04e4309-6708-42b0-b61b-f71b7037a26f" (UID: "e04e4309-6708-42b0-b61b-f71b7037a26f"). InnerVolumeSpecName "kube-api-access-kswhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:48:41.727551 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.727518 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e04e4309-6708-42b0-b61b-f71b7037a26f-extensions-socket-volume\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:48:41.727551 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:41.727544 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kswhr\" (UniqueName: \"kubernetes.io/projected/e04e4309-6708-42b0-b61b-f71b7037a26f-kube-api-access-kswhr\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:48:42.128464 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:42.128388 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" Jul 15 19:48:42.128906 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:42.128384 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg" event={"ID":"e04e4309-6708-42b0-b61b-f71b7037a26f","Type":"ContainerDied","Data":"efa91f4e821d6e333194349c64ec3bb3e0d281096e38d538a4f259eb00d899aa"} Jul 15 19:48:42.128906 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:42.128524 2573 scope.go:117] "RemoveContainer" containerID="57c32f0bb304950b65a4cc9a60d66147ca340d8fe600bf7b375fee168897af32" Jul 15 19:48:42.150396 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:42.150371 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg"] Jul 15 19:48:42.153885 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:42.153861 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wg5bg"] Jul 15 19:48:42.695591 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:48:42.695556 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e04e4309-6708-42b0-b61b-f71b7037a26f" path="/var/lib/kubelet/pods/e04e4309-6708-42b0-b61b-f71b7037a26f/volumes" Jul 15 19:49:39.689733 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.689691 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn"] Jul 15 19:49:39.690173 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.690159 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e04e4309-6708-42b0-b61b-f71b7037a26f" containerName="manager" Jul 15 19:49:39.690173 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.690175 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e04e4309-6708-42b0-b61b-f71b7037a26f" containerName="manager" Jul 15 19:49:39.690250 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.690239 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e04e4309-6708-42b0-b61b-f71b7037a26f" containerName="manager" Jul 15 19:49:39.693493 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.693477 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:39.696767 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.696745 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hfv28\"" Jul 15 19:49:39.715874 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.715846 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn"] Jul 15 19:49:39.852252 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.852219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac911a83-e947-49b8-a4ef-359dea0e4f5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smhhn\" (UID: \"ac911a83-e947-49b8-a4ef-359dea0e4f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:39.852426 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.852276 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhhn\" (UniqueName: \"kubernetes.io/projected/ac911a83-e947-49b8-a4ef-359dea0e4f5b-kube-api-access-llhhn\") pod \"kuadrant-operator-controller-manager-86878c769f-smhhn\" (UID: \"ac911a83-e947-49b8-a4ef-359dea0e4f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:39.953205 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.953119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhhn\" (UniqueName: \"kubernetes.io/projected/ac911a83-e947-49b8-a4ef-359dea0e4f5b-kube-api-access-llhhn\") pod \"kuadrant-operator-controller-manager-86878c769f-smhhn\" (UID: \"ac911a83-e947-49b8-a4ef-359dea0e4f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:39.953205 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.953190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac911a83-e947-49b8-a4ef-359dea0e4f5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smhhn\" (UID: \"ac911a83-e947-49b8-a4ef-359dea0e4f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:39.953564 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.953545 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac911a83-e947-49b8-a4ef-359dea0e4f5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smhhn\" (UID: \"ac911a83-e947-49b8-a4ef-359dea0e4f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:39.960940 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:39.960914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhhn\" (UniqueName: \"kubernetes.io/projected/ac911a83-e947-49b8-a4ef-359dea0e4f5b-kube-api-access-llhhn\") pod \"kuadrant-operator-controller-manager-86878c769f-smhhn\" (UID: \"ac911a83-e947-49b8-a4ef-359dea0e4f5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:40.003625 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:40.003582 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:40.141767 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:40.141741 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn"] Jul 15 19:49:40.143888 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:49:40.143862 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac911a83_e947_49b8_a4ef_359dea0e4f5b.slice/crio-cbd292aa1a29f1d1e0cbe5421118c47f023af029373b4366ffc1e20cb82f0568 WatchSource:0}: Error finding container cbd292aa1a29f1d1e0cbe5421118c47f023af029373b4366ffc1e20cb82f0568: Status 404 returned error can't find the container with id cbd292aa1a29f1d1e0cbe5421118c47f023af029373b4366ffc1e20cb82f0568 Jul 15 19:49:40.407413 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:40.407382 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" event={"ID":"ac911a83-e947-49b8-a4ef-359dea0e4f5b","Type":"ContainerStarted","Data":"76a784efe17e9d24a92316efcad7f665e2417645c4ef31132d55472781e13788"} Jul 15 19:49:40.407583 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:40.407430 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" event={"ID":"ac911a83-e947-49b8-a4ef-359dea0e4f5b","Type":"ContainerStarted","Data":"cbd292aa1a29f1d1e0cbe5421118c47f023af029373b4366ffc1e20cb82f0568"} Jul 15 19:49:40.407583 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:40.407523 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:49:40.427299 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:40.427242 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" podStartSLOduration=1.4272262150000001 podStartE2EDuration="1.427226215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:49:40.424629759 +0000 UTC m=+1262.285834112" watchObservedRunningTime="2026-07-15 19:49:40.427226215 +0000 UTC m=+1262.288430623" Jul 15 19:49:51.413947 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:49:51.413914 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smhhn" Jul 15 19:50:00.634822 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:50:00.634780 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:50:09.232968 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:50:09.232929 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:50:24.130589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:50:24.130552 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:50:34.026317 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:50:34.026284 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:50:44.227582 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:50:44.227545 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:50:52.441163 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:50:52.441128 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:51:01.928167 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:51:01.928069 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:51:12.836034 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:51:12.836002 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:51:22.836579 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:51:22.836538 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:51:32.331212 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:51:32.331177 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:51:45.336322 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:51:45.336286 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:51:54.537227 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:51:54.537191 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:52:10.533860 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:52:10.533825 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:52:20.639769 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:52:20.639730 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:52:32.233921 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:52:32.233838 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:52:42.835797 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:52:42.835761 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:53:00.236542 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:00.236503 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:53:09.634838 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:09.634795 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:53:19.231554 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:19.231521 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:53:29.331082 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:29.331049 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:53:38.686811 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:38.686777 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:53:38.688763 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:38.688741 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:53:38.695840 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:38.695790 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:53:38.698242 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:38.698222 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:53:38.733492 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:38.733467 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:53:56.936909 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:53:56.936871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:54:06.135699 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:54:06.135609 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:54:15.338624 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:54:15.338591 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:54:24.840898 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:54:24.840858 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:54:35.736119 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:54:35.736076 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:54:48.236332 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:54:48.236295 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:54:58.039269 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:54:58.039237 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:55:10.237777 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:55:10.237739 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:55:22.237725 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:55:22.237691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:55:33.638046 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:55:33.637956 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:55:46.682870 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:55:46.682834 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:55:57.133641 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:55:57.133604 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:56:04.646916 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.646883 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb"] Jul 15 19:56:04.650816 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.650781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.653186 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.653168 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-de95558c-openshift-default-dockercfg-dmqdr\"" Jul 15 19:56:04.665008 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.664985 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb"] Jul 15 19:56:04.835118 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-credential-socket\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835308 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-certs\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835308 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-socket\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835308 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835224 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-token\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835308 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-envoy\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835445 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqkj\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-kube-api-access-tbqkj\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835445 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istiod-ca-cert\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835445 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-data\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.835445 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.835418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-podinfo\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.936610 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-data\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.936610 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-podinfo\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.936853 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-credential-socket\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.936853 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-certs\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.936853 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-socket\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937047 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-token\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937047 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-envoy\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937047 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqkj\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-kube-api-access-tbqkj\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937047 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.936983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istiod-ca-cert\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937209 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.937077 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-certs\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937209 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.937118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-credential-socket\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937436 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.937417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-data\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.937521 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.937471 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-socket\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.938228 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.938205 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istiod-ca-cert\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.939379 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.939358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-envoy\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.939552 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.939526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-podinfo\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.947272 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.947248 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-token\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.947467 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.947447 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqkj\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-kube-api-access-tbqkj\") pod \"e2e-ait-de95558c-openshift-default-6945888c99-5lbqb\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:04.965171 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:04.965149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:05.113485 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.113455 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb"] Jul 15 19:56:05.115552 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:56:05.115526 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43b4ebb_1d8a_416a_9a0e_5be02b76a7a2.slice/crio-9f66576ed88fa549c330f2ce9d2fcbfae6d863e7da7a460303d29202bcfe6a17 WatchSource:0}: Error finding container 9f66576ed88fa549c330f2ce9d2fcbfae6d863e7da7a460303d29202bcfe6a17: Status 404 returned error can't find the container with id 9f66576ed88fa549c330f2ce9d2fcbfae6d863e7da7a460303d29202bcfe6a17 Jul 15 19:56:05.117563 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.117528 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:56:05.117658 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.117604 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:56:05.117658 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.117632 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:56:05.274465 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.274369 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" event={"ID":"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2","Type":"ContainerStarted","Data":"3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d"} Jul 15 19:56:05.274465 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.274433 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" event={"ID":"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2","Type":"ContainerStarted","Data":"9f66576ed88fa549c330f2ce9d2fcbfae6d863e7da7a460303d29202bcfe6a17"} Jul 15 19:56:05.293479 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.293425 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" podStartSLOduration=1.293403923 podStartE2EDuration="1.293403923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:56:05.292192215 +0000 UTC m=+1647.153396568" watchObservedRunningTime="2026-07-15 19:56:05.293403923 +0000 UTC m=+1647.154608277" Jul 15 19:56:05.966103 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:05.966061 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:06.970892 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:06.970863 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:07.285997 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:07.285923 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:07.286946 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:07.286924 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:09.930337 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:09.930304 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:56:18.342587 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:18.342551 2573 generic.go:356] "Generic (PLEG): container finished" podID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerID="3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d" exitCode=0 Jul 15 19:56:18.343034 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:18.342625 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" event={"ID":"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2","Type":"ContainerDied","Data":"3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d"} Jul 15 19:56:18.343118 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:18.343102 2573 scope.go:117] "RemoveContainer" containerID="3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d" Jul 15 19:56:18.344137 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:18.344107 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:56:18.344213 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:18.344192 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:56:18.344259 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:18.344246 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:56:19.350036 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:19.349999 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" event={"ID":"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2","Type":"ContainerStarted","Data":"c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43"} Jul 15 19:56:19.965875 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:19.965840 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:19.966843 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:19.966817 2573 patch_prober.go:28] interesting pod/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 15 19:56:19.966975 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:19.966867 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 15 19:56:20.965822 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:20.965776 2573 patch_prober.go:28] interesting pod/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 15 19:56:20.966212 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:20.965878 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 15 19:56:21.965984 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:21.965945 2573 patch_prober.go:28] interesting pod/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 15 19:56:21.966381 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:21.966005 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 15 19:56:22.285900 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:22.285781 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:22.965988 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:22.965947 2573 patch_prober.go:28] interesting pod/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 15 19:56:22.966396 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:22.966005 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 15 19:56:23.969518 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:23.969488 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:56:23.970494 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:56:23.970479 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:57:14.238759 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:14.238671 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:57:16.183890 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:16.183844 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb"] Jul 15 19:57:16.184359 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:16.184131 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" containerID="cri-o://c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43" gracePeriod=30 Jul 15 19:57:21.431058 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.431032 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:57:21.524495 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524470 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-data\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.524621 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524516 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-certs\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.524621 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524536 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-socket\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.524621 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524556 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istiod-ca-cert\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.524621 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524585 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbqkj\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-kube-api-access-tbqkj\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.524775 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524619 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-envoy\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.524847 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524829 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-podinfo\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.524904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524861 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:57:21.524904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524872 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:57:21.524904 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524880 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-credential-socket\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.525053 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.524986 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-token\") pod \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\" (UID: \"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2\") " Jul 15 19:57:21.525113 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.525080 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:57:21.525165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.525125 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:57:21.525338 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.525313 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.525338 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.525341 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.525710 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.525354 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.525710 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.525362 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.525710 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.525310 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:57:21.527036 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.527010 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:57:21.527135 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.527040 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-kube-api-access-tbqkj" (OuterVolumeSpecName: "kube-api-access-tbqkj") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "kube-api-access-tbqkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:57:21.527135 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.527102 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:57:21.527238 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.527222 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" (UID: "f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:57:21.626722 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.626687 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.626722 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.626714 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbqkj\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-kube-api-access-tbqkj\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.626722 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.626724 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.626722 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.626731 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.627033 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.626739 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:57:21.655576 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.655547 2573 generic.go:356] "Generic (PLEG): container finished" podID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerID="c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43" exitCode=0 Jul 15 19:57:21.655703 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.655592 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" event={"ID":"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2","Type":"ContainerDied","Data":"c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43"} Jul 15 19:57:21.655703 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.655597 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" Jul 15 19:57:21.655703 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.655635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb" event={"ID":"f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2","Type":"ContainerDied","Data":"9f66576ed88fa549c330f2ce9d2fcbfae6d863e7da7a460303d29202bcfe6a17"} Jul 15 19:57:21.655703 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.655659 2573 scope.go:117] "RemoveContainer" containerID="c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43" Jul 15 19:57:21.665987 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.665970 2573 scope.go:117] "RemoveContainer" containerID="3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d" Jul 15 19:57:21.674434 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.674415 2573 scope.go:117] "RemoveContainer" containerID="c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43" Jul 15 19:57:21.674686 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:57:21.674667 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43\": container with ID starting with c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43 not found: ID does not exist" containerID="c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43" Jul 15 19:57:21.674757 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.674698 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43"} err="failed to get container status \"c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43\": rpc error: code = NotFound desc = could not find container \"c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43\": container with ID starting with c574fcf2701861ed5921680421db928e61bce8ee7379ca10ca259a05beaa5f43 not found: ID does not exist" Jul 15 19:57:21.674757 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.674724 2573 scope.go:117] "RemoveContainer" containerID="3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d" Jul 15 19:57:21.675031 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:57:21.675009 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d\": container with ID starting with 3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d not found: ID does not exist" containerID="3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d" Jul 15 19:57:21.675127 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.675036 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d"} err="failed to get container status \"3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d\": rpc error: code = NotFound desc = could not find container \"3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d\": container with ID starting with 3c24888793e35184727bdf405becaa1fc74b2524304fb60af1f0d4d97005158d not found: ID does not exist" Jul 15 19:57:21.677561 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.677539 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb"] Jul 15 19:57:21.681891 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:21.681797 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-de95558c-openshift-default-6945888c99-5lbqb"] Jul 15 19:57:22.528846 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.528785 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8"] Jul 15 19:57:22.529302 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.529256 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" Jul 15 19:57:22.529302 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.529273 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" Jul 15 19:57:22.529395 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.529362 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" Jul 15 19:57:22.532453 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.532427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.534608 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.534590 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2791e51d-gw-openshift-default-dockercfg-57pfs\"" Jul 15 19:57:22.544240 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.544216 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8"] Jul 15 19:57:22.635164 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635130 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-certs\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635164 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-token\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635270 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-socket\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-envoy\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635340 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpzg\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-kube-api-access-jkpzg\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-credential-socket\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635557 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-podinfo\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635557 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-data\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.635557 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.635449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istiod-ca-cert\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.696986 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.696949 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" path="/var/lib/kubelet/pods/f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2/volumes" Jul 15 19:57:22.736061 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-credential-socket\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736163 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-podinfo\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736163 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-data\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736277 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istiod-ca-cert\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736277 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736237 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-certs\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736375 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-token\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-socket\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-envoy\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736531 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpzg\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-kube-api-access-jkpzg\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736531 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736479 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-credential-socket\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736711 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736677 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-socket\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736776 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736709 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-data\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.736956 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.736929 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-certs\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.737220 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.737197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istiod-ca-cert\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.738714 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.738693 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-envoy\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.738857 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.738840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-podinfo\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.752530 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.752508 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-token\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.752674 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.752512 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpzg\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-kube-api-access-jkpzg\") pod \"e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.844029 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.843936 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:22.976100 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.976074 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8"] Jul 15 19:57:22.977619 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:57:22.977591 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd486d2e_b8f1_4df6_be1d_2292324dbe66.slice/crio-723190c5a30dc86dd364e44631eb74ec6f2421d070c396c01d507b18a0e18b34 WatchSource:0}: Error finding container 723190c5a30dc86dd364e44631eb74ec6f2421d070c396c01d507b18a0e18b34: Status 404 returned error can't find the container with id 723190c5a30dc86dd364e44631eb74ec6f2421d070c396c01d507b18a0e18b34 Jul 15 19:57:22.979968 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.979936 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:57:22.980046 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.980001 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:57:22.980046 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:22.980030 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:57:23.670549 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:23.670518 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" event={"ID":"cd486d2e-b8f1-4df6-be1d-2292324dbe66","Type":"ContainerStarted","Data":"a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3"} Jul 15 19:57:23.670942 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:23.670568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" event={"ID":"cd486d2e-b8f1-4df6-be1d-2292324dbe66","Type":"ContainerStarted","Data":"723190c5a30dc86dd364e44631eb74ec6f2421d070c396c01d507b18a0e18b34"} Jul 15 19:57:23.689753 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:23.689704 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" podStartSLOduration=1.689687806 podStartE2EDuration="1.689687806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:57:23.688243843 +0000 UTC m=+1725.549448208" watchObservedRunningTime="2026-07-15 19:57:23.689687806 +0000 UTC m=+1725.550892158" Jul 15 19:57:23.844389 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:23.844348 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:24.849529 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:24.849491 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:25.681487 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:25.681418 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:25.682600 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:25.682579 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:27.840106 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:27.840072 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:57:35.742709 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:35.742677 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerID="a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3" exitCode=0 Jul 15 19:57:35.743128 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:35.742752 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" event={"ID":"cd486d2e-b8f1-4df6-be1d-2292324dbe66","Type":"ContainerDied","Data":"a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3"} Jul 15 19:57:35.743197 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:35.743182 2573 scope.go:117] "RemoveContainer" containerID="a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3" Jul 15 19:57:35.744352 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:35.744319 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:57:35.744431 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:35.744413 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:57:35.744467 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:35.744459 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:57:36.749354 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:36.749322 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" event={"ID":"cd486d2e-b8f1-4df6-be1d-2292324dbe66","Type":"ContainerStarted","Data":"422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09"} Jul 15 19:57:36.845059 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:36.845022 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:36.846104 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:36.846076 2573 patch_prober.go:28] interesting pod/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 15 19:57:36.846255 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:36.846135 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 15 19:57:37.845289 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:37.845252 2573 patch_prober.go:28] interesting pod/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 15 19:57:37.845663 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:37.845333 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 15 19:57:38.845286 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:38.845252 2573 patch_prober.go:28] interesting pod/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 15 19:57:38.845460 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:38.845318 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 15 19:57:39.845170 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:39.845134 2573 patch_prober.go:28] interesting pod/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 15 19:57:39.845351 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:39.845189 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 15 19:57:40.681773 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:40.681732 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:40.848494 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:40.848464 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:57:40.849602 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:57:40.849580 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:58:28.496166 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:28.496131 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8"] Jul 15 19:58:28.496641 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:28.496343 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" containerID="cri-o://422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09" gracePeriod=30 Jul 15 19:58:31.431595 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:31.431510 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:58:33.757234 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.757173 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:58:33.888908 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.888870 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-credential-socket\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889088 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.888916 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkpzg\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-kube-api-access-jkpzg\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889088 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.888987 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-token\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889088 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889011 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-data\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889088 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889046 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-podinfo\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889088 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889075 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istiod-ca-cert\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889153 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-socket\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889187 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-certs\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889188 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:58:33.889361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889215 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-envoy\") pod \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\" (UID: \"cd486d2e-b8f1-4df6-be1d-2292324dbe66\") " Jul 15 19:58:33.889361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889357 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:58:33.889596 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889540 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.889596 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889560 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.889704 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889600 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:58:33.889763 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889710 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:58:33.889885 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.889862 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:58:33.891678 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.891650 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:58:33.891789 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.891767 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-kube-api-access-jkpzg" (OuterVolumeSpecName: "kube-api-access-jkpzg") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "kube-api-access-jkpzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:58:33.891915 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.891896 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:58:33.891915 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.891898 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd486d2e-b8f1-4df6-be1d-2292324dbe66" (UID: "cd486d2e-b8f1-4df6-be1d-2292324dbe66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:58:33.990036 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.990007 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.990036 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.990035 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.990233 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.990045 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.990233 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.990057 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpzg\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-kube-api-access-jkpzg\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.990233 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.990069 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.990233 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.990080 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:33.990233 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:33.990090 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd486d2e-b8f1-4df6-be1d-2292324dbe66-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:58:34.028045 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.027959 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerID="422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09" exitCode=0 Jul 15 19:58:34.028045 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.028018 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" Jul 15 19:58:34.028265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.028046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" event={"ID":"cd486d2e-b8f1-4df6-be1d-2292324dbe66","Type":"ContainerDied","Data":"422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09"} Jul 15 19:58:34.028265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.028112 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8" event={"ID":"cd486d2e-b8f1-4df6-be1d-2292324dbe66","Type":"ContainerDied","Data":"723190c5a30dc86dd364e44631eb74ec6f2421d070c396c01d507b18a0e18b34"} Jul 15 19:58:34.028265 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.028142 2573 scope.go:117] "RemoveContainer" containerID="422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09" Jul 15 19:58:34.038137 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.038115 2573 scope.go:117] "RemoveContainer" containerID="a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3" Jul 15 19:58:34.049165 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.049144 2573 scope.go:117] "RemoveContainer" containerID="422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09" Jul 15 19:58:34.049253 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.049225 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8"] Jul 15 19:58:34.049461 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:58:34.049442 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09\": container with ID starting with 422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09 not found: ID does not exist" containerID="422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09" Jul 15 19:58:34.049523 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.049470 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09"} err="failed to get container status \"422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09\": rpc error: code = NotFound desc = could not find container \"422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09\": container with ID starting with 422ecd39997ef40eb01656688f300c906fd69386044304299f913148d9751b09 not found: ID does not exist" Jul 15 19:58:34.049523 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.049488 2573 scope.go:117] "RemoveContainer" containerID="a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3" Jul 15 19:58:34.050037 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:58:34.049987 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3\": container with ID starting with a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3 not found: ID does not exist" containerID="a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3" Jul 15 19:58:34.050037 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.050023 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3"} err="failed to get container status \"a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3\": rpc error: code = NotFound desc = could not find container \"a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3\": container with ID starting with a30ee4a1bf193332c104362da236da93e84f2f64be39d92dcc8f12b6e48103e3 not found: ID does not exist" Jul 15 19:58:34.052761 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.052740 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2791e51d-gw-openshift-default-8d6547c9f-5bkm8"] Jul 15 19:58:34.696283 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:34.696252 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" path="/var/lib/kubelet/pods/cd486d2e-b8f1-4df6-be1d-2292324dbe66/volumes" Jul 15 19:58:38.716941 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:38.716899 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:58:38.720547 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:38.720526 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 19:58:38.724676 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:38.724657 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:58:38.728332 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:38.728313 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 19:58:40.242260 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242230 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x"] Jul 15 19:58:40.242646 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242595 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" Jul 15 19:58:40.242646 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242610 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" Jul 15 19:58:40.242646 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242623 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" Jul 15 19:58:40.242646 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242631 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" Jul 15 19:58:40.242844 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242760 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" Jul 15 19:58:40.242844 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242771 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43b4ebb-1d8a-416a-9a0e-5be02b76a7a2" containerName="istio-proxy" Jul 15 19:58:40.242844 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.242780 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" Jul 15 19:58:40.245921 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.245900 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.248495 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.248476 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-37d9d895-openshift-default-dockercfg-m5svw\"" Jul 15 19:58:40.267674 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.267639 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x"] Jul 15 19:58:40.334191 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-token\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334376 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-envoy\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334376 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-certs\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334376 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-socket\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334376 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-data\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334557 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-credential-socket\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334557 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92996f28-af12-45f9-8a5b-519e8dc9bab3-istiod-ca-cert\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334557 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7hk\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-kube-api-access-vp7hk\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.334557 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.334516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-podinfo\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435190 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-podinfo\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435376 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-token\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435429 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-envoy\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435488 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-certs\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435540 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-socket\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435540 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-data\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435644 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-credential-socket\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435644 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92996f28-af12-45f9-8a5b-519e8dc9bab3-istiod-ca-cert\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435644 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7hk\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-kube-api-access-vp7hk\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435895 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-certs\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.435974 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435920 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-data\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.436031 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.435997 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-credential-socket\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.436255 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.436133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-socket\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.436311 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.436263 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92996f28-af12-45f9-8a5b-519e8dc9bab3-istiod-ca-cert\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.437767 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.437748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-envoy\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.438054 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.438038 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-podinfo\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.443452 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.443433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-token\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.443577 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.443557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7hk\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-kube-api-access-vp7hk\") pod \"e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.557651 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.557553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:40.699734 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.699709 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x"] Jul 15 19:58:40.702071 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:58:40.702033 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92996f28_af12_45f9_8a5b_519e8dc9bab3.slice/crio-ce7055553365e916460ef670afcd0cf39bfcbccaad26cd2cfc8fbad8b0a76bdc WatchSource:0}: Error finding container ce7055553365e916460ef670afcd0cf39bfcbccaad26cd2cfc8fbad8b0a76bdc: Status 404 returned error can't find the container with id ce7055553365e916460ef670afcd0cf39bfcbccaad26cd2cfc8fbad8b0a76bdc Jul 15 19:58:40.704451 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.704416 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:58:40.704560 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.704494 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:58:40.704560 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:40.704542 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:58:41.065157 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:41.065118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" event={"ID":"92996f28-af12-45f9-8a5b-519e8dc9bab3","Type":"ContainerStarted","Data":"3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd"} Jul 15 19:58:41.065330 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:41.065182 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" event={"ID":"92996f28-af12-45f9-8a5b-519e8dc9bab3","Type":"ContainerStarted","Data":"ce7055553365e916460ef670afcd0cf39bfcbccaad26cd2cfc8fbad8b0a76bdc"} Jul 15 19:58:41.088361 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:41.088302 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" podStartSLOduration=1.088281969 podStartE2EDuration="1.088281969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:58:41.087529857 +0000 UTC m=+1802.948734230" watchObservedRunningTime="2026-07-15 19:58:41.088281969 +0000 UTC m=+1802.949486326" Jul 15 19:58:41.558011 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:41.557976 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:41.562844 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:41.562797 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:42.069736 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:42.069707 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:42.070651 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:42.070630 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:43.930328 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:43.930288 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:58:52.141782 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:52.141675 2573 generic.go:356] "Generic (PLEG): container finished" podID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerID="3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd" exitCode=0 Jul 15 19:58:52.141782 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:52.141754 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" event={"ID":"92996f28-af12-45f9-8a5b-519e8dc9bab3","Type":"ContainerDied","Data":"3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd"} Jul 15 19:58:52.142327 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:52.142175 2573 scope.go:117] "RemoveContainer" containerID="3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd" Jul 15 19:58:52.143226 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:52.143195 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:58:52.143350 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:52.143284 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:58:52.143420 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:52.143346 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:58:53.157088 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:53.157050 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" event={"ID":"92996f28-af12-45f9-8a5b-519e8dc9bab3","Type":"ContainerStarted","Data":"bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71"} Jul 15 19:58:53.558767 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:53.558678 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:53.559923 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:53.559897 2573 patch_prober.go:28] interesting pod/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 15 19:58:53.560082 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:53.559961 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 15 19:58:54.558908 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:54.558868 2573 patch_prober.go:28] interesting pod/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 15 19:58:54.559326 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:54.558934 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 15 19:58:55.558943 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:55.558906 2573 patch_prober.go:28] interesting pod/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 15 19:58:55.559370 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:55.558962 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 15 19:58:56.558456 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:56.558418 2573 patch_prober.go:28] interesting pod/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 15 19:58:56.558650 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:56.558477 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 15 19:58:57.069889 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:57.069840 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:57.562633 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:57.562603 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:58:57.563602 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:58:57.563582 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:59:50.037095 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:50.037059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 19:59:53.213299 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:53.213266 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x"] Jul 15 19:59:53.213706 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:53.213471 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" containerID="cri-o://bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71" gracePeriod=30 Jul 15 19:59:57.565148 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:57.565110 2573 patch_prober.go:28] interesting pod/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 19:59:57.565527 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:57.565191 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:59:58.470555 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.470531 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:59:58.479270 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.479198 2573 generic.go:356] "Generic (PLEG): container finished" podID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerID="bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71" exitCode=0 Jul 15 19:59:58.479401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.479298 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" Jul 15 19:59:58.479401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.479327 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" event={"ID":"92996f28-af12-45f9-8a5b-519e8dc9bab3","Type":"ContainerDied","Data":"bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71"} Jul 15 19:59:58.479401 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.479385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x" event={"ID":"92996f28-af12-45f9-8a5b-519e8dc9bab3","Type":"ContainerDied","Data":"ce7055553365e916460ef670afcd0cf39bfcbccaad26cd2cfc8fbad8b0a76bdc"} Jul 15 19:59:58.479533 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.479417 2573 scope.go:117] "RemoveContainer" containerID="bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71" Jul 15 19:59:58.488611 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.488590 2573 scope.go:117] "RemoveContainer" containerID="3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd" Jul 15 19:59:58.501248 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.501225 2573 scope.go:117] "RemoveContainer" containerID="bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71" Jul 15 19:59:58.501612 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:59:58.501578 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71\": container with ID starting with bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71 not found: ID does not exist" containerID="bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71" Jul 15 19:59:58.501703 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.501622 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71"} err="failed to get container status \"bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71\": rpc error: code = NotFound desc = could not find container \"bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71\": container with ID starting with bbf7e0f74e5991ed6e006e15b766a9a2732ca5992568d07622914c78c5a0bd71 not found: ID does not exist" Jul 15 19:59:58.501703 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.501649 2573 scope.go:117] "RemoveContainer" containerID="3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd" Jul 15 19:59:58.501985 ip-10-0-142-40 kubenswrapper[2573]: E0715 19:59:58.501964 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd\": container with ID starting with 3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd not found: ID does not exist" containerID="3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd" Jul 15 19:59:58.502074 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.501990 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd"} err="failed to get container status \"3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd\": rpc error: code = NotFound desc = could not find container \"3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd\": container with ID starting with 3b79f49770da6fe7a581fa056276ad48977a168d9028d8e2588f99c83f48bfcd not found: ID does not exist" Jul 15 19:59:58.567510 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567484 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-envoy\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567518 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-credential-socket\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567557 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp7hk\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-kube-api-access-vp7hk\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567587 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-podinfo\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567623 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-socket\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567652 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-certs\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567676 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92996f28-af12-45f9-8a5b-519e8dc9bab3-istiod-ca-cert\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567702 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-data\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.567967 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.567730 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-token\") pod \"92996f28-af12-45f9-8a5b-519e8dc9bab3\" (UID: \"92996f28-af12-45f9-8a5b-519e8dc9bab3\") " Jul 15 19:59:58.568570 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.568068 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:58.568570 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.568135 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:58.568570 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.568221 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92996f28-af12-45f9-8a5b-519e8dc9bab3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:59:58.568570 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.568199 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:58.568795 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.568681 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-data" (OuterVolumeSpecName: "istio-data") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:58.570041 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.570014 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 19:59:58.570160 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.570118 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-token" (OuterVolumeSpecName: "istio-token") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:59:58.570317 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.570301 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-kube-api-access-vp7hk" (OuterVolumeSpecName: "kube-api-access-vp7hk") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "kube-api-access-vp7hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:59:58.570317 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.570302 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92996f28-af12-45f9-8a5b-519e8dc9bab3" (UID: "92996f28-af12-45f9-8a5b-519e8dc9bab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:58.669131 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669098 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669131 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669121 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669131 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669133 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669131 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669142 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7hk\" (UniqueName: \"kubernetes.io/projected/92996f28-af12-45f9-8a5b-519e8dc9bab3-kube-api-access-vp7hk\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669411 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669152 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669411 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669162 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669411 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669169 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669411 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669177 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92996f28-af12-45f9-8a5b-519e8dc9bab3-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.669411 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.669184 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92996f28-af12-45f9-8a5b-519e8dc9bab3-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 19:59:58.795022 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.794937 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x"] Jul 15 19:59:58.798455 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:58.798432 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-37d9d895-openshift-default-58894b7d9c-4px5x"] Jul 15 19:59:59.329189 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329151 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn"] Jul 15 19:59:59.329543 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329531 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" Jul 15 19:59:59.329589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329545 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" Jul 15 19:59:59.329589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329553 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" Jul 15 19:59:59.329589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329559 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd486d2e-b8f1-4df6-be1d-2292324dbe66" containerName="istio-proxy" Jul 15 19:59:59.329589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329567 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" Jul 15 19:59:59.329589 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329572 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" Jul 15 19:59:59.329770 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329655 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" Jul 15 19:59:59.329770 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.329663 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" containerName="istio-proxy" Jul 15 19:59:59.334226 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.334201 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.336319 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.336299 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-033c5f64-openshift-default-dockercfg-lbkfj\"" Jul 15 19:59:59.346027 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.346002 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn"] Jul 15 19:59:59.475278 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2890bac-0f1b-4627-86b4-44736f9e9cff-istiod-ca-cert\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475278 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-credential-socket\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475300 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-data\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-podinfo\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-envoy\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwfhv\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-kube-api-access-mwfhv\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-certs\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475607 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-token\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.475835 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.475623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-socket\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577032 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.576989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-envoy\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwfhv\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-kube-api-access-mwfhv\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-certs\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-token\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-socket\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2890bac-0f1b-4627-86b4-44736f9e9cff-istiod-ca-cert\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-credential-socket\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-data\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.577525 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-podinfo\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.578012 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.577532 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-certs\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.578095 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.578073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-credential-socket\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.578336 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.578315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-data\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.578416 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.578318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2890bac-0f1b-4627-86b4-44736f9e9cff-istiod-ca-cert\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.578591 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.578567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-socket\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.580069 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.579994 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-envoy\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.580191 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.580140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-podinfo\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.586443 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.586416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-token\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.586625 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.586603 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwfhv\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-kube-api-access-mwfhv\") pod \"e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.645359 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.645330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 19:59:59.789569 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.789546 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn"] Jul 15 19:59:59.791648 ip-10-0-142-40 kubenswrapper[2573]: W0715 19:59:59.791618 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2890bac_0f1b_4627_86b4_44736f9e9cff.slice/crio-7b7d1e715cc7f928777f6f96f49b8dbd0b7c5278262941a87b3d7f372dc9d458 WatchSource:0}: Error finding container 7b7d1e715cc7f928777f6f96f49b8dbd0b7c5278262941a87b3d7f372dc9d458: Status 404 returned error can't find the container with id 7b7d1e715cc7f928777f6f96f49b8dbd0b7c5278262941a87b3d7f372dc9d458 Jul 15 19:59:59.793692 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.793639 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:59:59.793793 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.793732 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 19:59:59.793793 ip-10-0-142-40 kubenswrapper[2573]: I0715 19:59:59.793764 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:00:00.493524 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:00.493492 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" event={"ID":"f2890bac-0f1b-4627-86b4-44736f9e9cff","Type":"ContainerStarted","Data":"074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0"} Jul 15 20:00:00.493705 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:00.493542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" event={"ID":"f2890bac-0f1b-4627-86b4-44736f9e9cff","Type":"ContainerStarted","Data":"7b7d1e715cc7f928777f6f96f49b8dbd0b7c5278262941a87b3d7f372dc9d458"} Jul 15 20:00:00.513067 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:00.513009 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" podStartSLOduration=1.512991713 podStartE2EDuration="1.512991713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:00:00.511552066 +0000 UTC m=+1882.372756436" watchObservedRunningTime="2026-07-15 20:00:00.512991713 +0000 UTC m=+1882.374196066" Jul 15 20:00:00.646459 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:00.646426 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:00:00.651478 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:00.651458 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:00:00.696364 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:00.696329 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92996f28-af12-45f9-8a5b-519e8dc9bab3" path="/var/lib/kubelet/pods/92996f28-af12-45f9-8a5b-519e8dc9bab3/volumes" Jul 15 20:00:01.498323 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:01.498292 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:00:01.499409 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:01.499374 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:00:04.037618 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:04.037581 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:00:12.555136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:12.555023 2573 generic.go:356] "Generic (PLEG): container finished" podID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerID="074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0" exitCode=0 Jul 15 20:00:12.555136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:12.555106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" event={"ID":"f2890bac-0f1b-4627-86b4-44736f9e9cff","Type":"ContainerDied","Data":"074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0"} Jul 15 20:00:12.556123 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:12.555625 2573 scope.go:117] "RemoveContainer" containerID="074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0" Jul 15 20:00:12.556731 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:12.556702 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:00:12.556792 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:12.556778 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:00:12.556870 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:12.556839 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:00:13.564189 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:13.564159 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" event={"ID":"f2890bac-0f1b-4627-86b4-44736f9e9cff","Type":"ContainerStarted","Data":"3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df"} Jul 15 20:00:13.645583 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:13.645553 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:00:13.646923 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:13.646890 2573 patch_prober.go:28] interesting pod/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:00:13.647062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:13.646968 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 20:00:14.646183 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:14.646153 2573 patch_prober.go:28] interesting pod/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:00:14.646645 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:14.646212 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 20:00:15.646756 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:15.646719 2573 patch_prober.go:28] interesting pod/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:00:15.647192 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:15.646774 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 20:00:16.498554 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:16.498523 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:00:16.646687 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:16.646650 2573 patch_prober.go:28] interesting pod/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:00:16.646890 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:16.646707 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 20:00:17.645979 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:17.645944 2573 patch_prober.go:28] interesting pod/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:00:17.646167 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:17.645997 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 20:00:18.649751 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:18.649714 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:00:18.650727 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:00:18.650706 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:01:18.339011 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:18.338973 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:01:20.188350 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:20.188314 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn"] Jul 15 20:01:20.188733 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:20.188562 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" containerID="cri-o://3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df" gracePeriod=30 Jul 15 20:01:25.446053 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.446026 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:01:25.572103 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572067 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-credential-socket\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572273 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572110 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-envoy\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572273 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572139 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-certs\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572273 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572174 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-data\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572273 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572198 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwfhv\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-kube-api-access-mwfhv\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572273 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572253 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-socket\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572538 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572312 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-token\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572538 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572375 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2890bac-0f1b-4627-86b4-44736f9e9cff-istiod-ca-cert\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572538 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572400 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:01:25.572538 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572411 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-podinfo\") pod \"f2890bac-0f1b-4627-86b4-44736f9e9cff\" (UID: \"f2890bac-0f1b-4627-86b4-44736f9e9cff\") " Jul 15 20:01:25.572538 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572444 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:01:25.572841 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572643 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:01:25.572931 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.572912 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.573127 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.573107 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.573238 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.573129 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.573652 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.573520 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2890bac-0f1b-4627-86b4-44736f9e9cff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:01:25.574329 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.574301 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:01:25.574930 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.574909 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:01:25.575013 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.574997 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-kube-api-access-mwfhv" (OuterVolumeSpecName: "kube-api-access-mwfhv") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "kube-api-access-mwfhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:01:25.575350 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.575319 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:01:25.575654 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.575636 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2890bac-0f1b-4627-86b4-44736f9e9cff" (UID: "f2890bac-0f1b-4627-86b4-44736f9e9cff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:01:25.673648 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.673613 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.673648 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.673645 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2890bac-0f1b-4627-86b4-44736f9e9cff-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.673648 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.673655 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.673950 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.673663 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.673950 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.673671 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2890bac-0f1b-4627-86b4-44736f9e9cff-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.673950 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.673679 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwfhv\" (UniqueName: \"kubernetes.io/projected/f2890bac-0f1b-4627-86b4-44736f9e9cff-kube-api-access-mwfhv\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:01:25.927420 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.927384 2573 generic.go:356] "Generic (PLEG): container finished" podID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerID="3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df" exitCode=0 Jul 15 20:01:25.927617 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.927432 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" Jul 15 20:01:25.927617 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.927434 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" event={"ID":"f2890bac-0f1b-4627-86b4-44736f9e9cff","Type":"ContainerDied","Data":"3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df"} Jul 15 20:01:25.927617 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.927495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn" event={"ID":"f2890bac-0f1b-4627-86b4-44736f9e9cff","Type":"ContainerDied","Data":"7b7d1e715cc7f928777f6f96f49b8dbd0b7c5278262941a87b3d7f372dc9d458"} Jul 15 20:01:25.927617 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.927525 2573 scope.go:117] "RemoveContainer" containerID="3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df" Jul 15 20:01:25.937209 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.937190 2573 scope.go:117] "RemoveContainer" containerID="074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0" Jul 15 20:01:25.945471 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.945414 2573 scope.go:117] "RemoveContainer" containerID="3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df" Jul 15 20:01:25.945930 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:01:25.945896 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df\": container with ID starting with 3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df not found: ID does not exist" containerID="3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df" Jul 15 20:01:25.946013 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.945939 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df"} err="failed to get container status \"3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df\": rpc error: code = NotFound desc = could not find container \"3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df\": container with ID starting with 3ebefec7f314c525bf7298402917813bfcd191de97ec793cd999132bdf5f32df not found: ID does not exist" Jul 15 20:01:25.946013 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.945963 2573 scope.go:117] "RemoveContainer" containerID="074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0" Jul 15 20:01:25.946256 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:01:25.946240 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0\": container with ID starting with 074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0 not found: ID does not exist" containerID="074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0" Jul 15 20:01:25.946318 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.946260 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0"} err="failed to get container status \"074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0\": rpc error: code = NotFound desc = could not find container \"074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0\": container with ID starting with 074a40eb12ff9c5adc8472a89a37b11485b9992cdccdf355c6c77eabb5ea0db0 not found: ID does not exist" Jul 15 20:01:25.947290 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.947268 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn"] Jul 15 20:01:25.950638 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:25.950617 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-033c5f64-openshift-default-7c599f6b47-8lgcn"] Jul 15 20:01:26.695513 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:01:26.695479 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" path="/var/lib/kubelet/pods/f2890bac-0f1b-4627-86b4-44736f9e9cff/volumes" Jul 15 20:02:52.440057 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:02:52.440025 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:03:01.734762 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:01.734678 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:03:03.570219 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.570187 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr"] Jul 15 20:03:03.570620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.570580 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" Jul 15 20:03:03.570620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.570591 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" Jul 15 20:03:03.570620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.570600 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" Jul 15 20:03:03.570620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.570605 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" Jul 15 20:03:03.570751 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.570662 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" Jul 15 20:03:03.570751 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.570671 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2890bac-0f1b-4627-86b4-44736f9e9cff" containerName="istio-proxy" Jul 15 20:03:03.573764 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.573747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.575928 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.575899 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3a3ed256-openshift-default-dockercfg-kflrh\"" Jul 15 20:03:03.589324 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.589296 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr"] Jul 15 20:03:03.725046 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-token\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725244 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-credential-socket\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725244 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725086 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-data\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725244 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-certs\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725244 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9swb\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-kube-api-access-p9swb\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725244 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istiod-ca-cert\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725445 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-socket\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725445 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-envoy\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.725445 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.725322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-podinfo\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.826743 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.826649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-socket\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.826743 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.826683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-envoy\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.826743 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.826701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-podinfo\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.826750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-token\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.826776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-credential-socket\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.826792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-data\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.826868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-certs\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827281 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.827133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9swb\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-kube-api-access-p9swb\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827281 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.827174 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-socket\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827281 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.827180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istiod-ca-cert\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827281 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.827234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-certs\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827465 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.827324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-credential-socket\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.827511 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.827477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-data\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.828102 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.828079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istiod-ca-cert\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.829454 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.829433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-envoy\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.829561 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.829526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-podinfo\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.834823 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.834786 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9swb\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-kube-api-access-p9swb\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.834918 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.834837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-token\") pod \"e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:03.886817 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:03.886772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:04.036346 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.036315 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr"] Jul 15 20:03:04.039603 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:03:04.039577 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4312a8_e16f_4e92_a2f8_8bc5ecf024ac.slice/crio-3a77e1a90a217bdce2b95b4d3650d9e6702c86876a0824225d8dbd099a20b0b9 WatchSource:0}: Error finding container 3a77e1a90a217bdce2b95b4d3650d9e6702c86876a0824225d8dbd099a20b0b9: Status 404 returned error can't find the container with id 3a77e1a90a217bdce2b95b4d3650d9e6702c86876a0824225d8dbd099a20b0b9 Jul 15 20:03:04.041966 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.041926 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:03:04.042043 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.042016 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:03:04.042081 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.042056 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:03:04.412356 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.412318 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" event={"ID":"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac","Type":"ContainerStarted","Data":"54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7"} Jul 15 20:03:04.412525 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.412383 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" event={"ID":"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac","Type":"ContainerStarted","Data":"3a77e1a90a217bdce2b95b4d3650d9e6702c86876a0824225d8dbd099a20b0b9"} Jul 15 20:03:04.433646 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.433596 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" podStartSLOduration=1.433579594 podStartE2EDuration="1.433579594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:03:04.431383084 +0000 UTC m=+2066.292587452" watchObservedRunningTime="2026-07-15 20:03:04.433579594 +0000 UTC m=+2066.294783946" Jul 15 20:03:04.887568 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:04.887474 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:05.893033 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:05.892999 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:06.423263 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:06.423238 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:06.424219 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:06.424202 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:10.027445 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:10.027415 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:03:18.483554 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:18.483517 2573 generic.go:356] "Generic (PLEG): container finished" podID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerID="54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7" exitCode=0 Jul 15 20:03:18.484013 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:18.483594 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" event={"ID":"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac","Type":"ContainerDied","Data":"54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7"} Jul 15 20:03:18.484058 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:18.484016 2573 scope.go:117] "RemoveContainer" containerID="54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7" Jul 15 20:03:18.485155 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:18.485123 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:03:18.485226 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:18.485208 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:03:18.485268 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:18.485242 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:03:19.491391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:19.491293 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" event={"ID":"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac","Type":"ContainerStarted","Data":"7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933"} Jul 15 20:03:19.888007 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:19.887903 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:19.889198 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:19.889164 2573 patch_prober.go:28] interesting pod/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:03:19.889341 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:19.889224 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 20:03:20.887375 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:20.887327 2573 patch_prober.go:28] interesting pod/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:03:20.887933 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:20.887399 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 20:03:21.423431 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:21.423397 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:21.887526 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:21.887418 2573 patch_prober.go:28] interesting pod/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:03:21.887526 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:21.887505 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 20:03:22.888255 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:22.888216 2573 patch_prober.go:28] interesting pod/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:03:22.888943 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:22.888911 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 20:03:23.891230 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:23.891202 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:23.892157 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:23.892138 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:30.538492 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:30.538450 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:03:38.746296 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:38.746262 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 20:03:38.751433 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:38.751388 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 20:03:38.753849 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:38.753830 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 20:03:38.758876 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:38.758860 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 20:03:39.857427 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:39.857384 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr"] Jul 15 20:03:39.857939 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:39.857670 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" containerID="cri-o://7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933" gracePeriod=30 Jul 15 20:03:45.130180 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.130117 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:45.208219 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208179 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-podinfo\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208386 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208234 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-envoy\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208386 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208275 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-certs\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208386 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208324 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-token\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208386 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208351 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-credential-socket\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208386 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208380 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-data\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208656 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208416 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9swb\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-kube-api-access-p9swb\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208656 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208448 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-socket\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208656 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208489 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istiod-ca-cert\") pod \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\" (UID: \"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac\") " Jul 15 20:03:45.208656 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208609 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:03:45.208892 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208706 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:03:45.208892 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.208847 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:03:45.209064 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.209030 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:03:45.209199 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.209123 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.209199 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.209139 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.209199 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.209153 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.209199 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.209165 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.209573 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.209549 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:03:45.210655 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.210623 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:03:45.210940 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.210916 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-kube-api-access-p9swb" (OuterVolumeSpecName: "kube-api-access-p9swb") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "kube-api-access-p9swb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:03:45.211117 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.211092 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:03:45.211117 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.211102 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" (UID: "bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:03:45.310365 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.310317 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.310365 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.310361 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9swb\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-kube-api-access-p9swb\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.310365 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.310371 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.310365 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.310379 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.310365 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.310387 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:03:45.622697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.622658 2573 generic.go:356] "Generic (PLEG): container finished" podID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerID="7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933" exitCode=0 Jul 15 20:03:45.622890 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.622713 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" Jul 15 20:03:45.622890 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.622717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" event={"ID":"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac","Type":"ContainerDied","Data":"7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933"} Jul 15 20:03:45.622890 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.622771 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr" event={"ID":"bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac","Type":"ContainerDied","Data":"3a77e1a90a217bdce2b95b4d3650d9e6702c86876a0824225d8dbd099a20b0b9"} Jul 15 20:03:45.622890 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.622818 2573 scope.go:117] "RemoveContainer" containerID="7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933" Jul 15 20:03:45.632609 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.632589 2573 scope.go:117] "RemoveContainer" containerID="54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7" Jul 15 20:03:45.641157 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.641134 2573 scope.go:117] "RemoveContainer" containerID="7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933" Jul 15 20:03:45.641553 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:03:45.641510 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933\": container with ID starting with 7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933 not found: ID does not exist" containerID="7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933" Jul 15 20:03:45.641656 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.641567 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933"} err="failed to get container status \"7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933\": rpc error: code = NotFound desc = could not find container \"7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933\": container with ID starting with 7bc8a87eb580da9d1132232dc0d21a3a121e32eb980cb6513a1e2d222743b933 not found: ID does not exist" Jul 15 20:03:45.641656 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.641588 2573 scope.go:117] "RemoveContainer" containerID="54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7" Jul 15 20:03:45.641870 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:03:45.641852 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7\": container with ID starting with 54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7 not found: ID does not exist" containerID="54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7" Jul 15 20:03:45.641943 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.641879 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7"} err="failed to get container status \"54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7\": rpc error: code = NotFound desc = could not find container \"54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7\": container with ID starting with 54c021b895ad078e1836fa543a3c4f5bc62cf577920e3cf6895123820b5f25b7 not found: ID does not exist" Jul 15 20:03:45.642861 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.642840 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr"] Jul 15 20:03:45.646725 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:45.646707 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3a3ed256-openshift-default-6747b66cf-mq5wr"] Jul 15 20:03:46.696775 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:46.696742 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" path="/var/lib/kubelet/pods/bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac/volumes" Jul 15 20:03:59.716791 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.716756 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr"] Jul 15 20:03:59.717491 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.717166 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" Jul 15 20:03:59.717491 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.717179 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" Jul 15 20:03:59.717491 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.717188 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" Jul 15 20:03:59.717491 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.717194 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" Jul 15 20:03:59.717491 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.717285 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" Jul 15 20:03:59.721864 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.721845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.723919 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.723899 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-12e2475c-openshift-default-dockercfg-bx9dk\"" Jul 15 20:03:59.731296 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.731272 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr"] Jul 15 20:03:59.838199 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-istio-token\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838371 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54889e9d-448b-4243-89aa-760ceb26766c-istiod-ca-cert\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838371 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-certs\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838371 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838299 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-data\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838481 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdr9m\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-kube-api-access-xdr9m\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838481 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54889e9d-448b-4243-89aa-760ceb26766c-istio-podinfo\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838481 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-socket\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838581 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-credential-socket\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.838581 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.838534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-envoy\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939118 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-istio-token\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939118 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54889e9d-448b-4243-89aa-760ceb26766c-istiod-ca-cert\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-certs\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-data\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdr9m\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-kube-api-access-xdr9m\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54889e9d-448b-4243-89aa-760ceb26766c-istio-podinfo\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939600 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-socket\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939600 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-credential-socket\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939600 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-certs\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939600 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-envoy\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939895 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-data\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939895 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-socket\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939994 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939913 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-credential-socket\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.939994 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.939923 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54889e9d-448b-4243-89aa-760ceb26766c-istiod-ca-cert\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.941924 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.941901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-envoy\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.942085 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.942069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54889e9d-448b-4243-89aa-760ceb26766c-istio-podinfo\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.947095 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.947068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-istio-token\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:03:59.947336 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:03:59.947317 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdr9m\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-kube-api-access-xdr9m\") pod \"e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:00.033139 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.033047 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:00.180502 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.180453 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr"] Jul 15 20:04:00.182088 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:04:00.182057 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54889e9d_448b_4243_89aa_760ceb26766c.slice/crio-a653388565fa9f0d7280ba0d163f7a5297f4a24e38c5893e7883adfb38032bfa WatchSource:0}: Error finding container a653388565fa9f0d7280ba0d163f7a5297f4a24e38c5893e7883adfb38032bfa: Status 404 returned error can't find the container with id a653388565fa9f0d7280ba0d163f7a5297f4a24e38c5893e7883adfb38032bfa Jul 15 20:04:00.184317 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.184278 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:04:00.184407 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.184368 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:04:00.184448 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.184405 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:04:00.703940 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.703897 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" event={"ID":"54889e9d-448b-4243-89aa-760ceb26766c","Type":"ContainerStarted","Data":"d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec"} Jul 15 20:04:00.704139 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.703949 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" event={"ID":"54889e9d-448b-4243-89aa-760ceb26766c","Type":"ContainerStarted","Data":"a653388565fa9f0d7280ba0d163f7a5297f4a24e38c5893e7883adfb38032bfa"} Jul 15 20:04:00.722243 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:00.722196 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" podStartSLOduration=1.7221788660000001 podStartE2EDuration="1.722178866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:04:00.721362211 +0000 UTC m=+2122.582566565" watchObservedRunningTime="2026-07-15 20:04:00.722178866 +0000 UTC m=+2122.583383221" Jul 15 20:04:01.033557 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:01.033468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:01.038087 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:01.038060 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:01.709191 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:01.709160 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:01.710057 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:01.710037 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:08.130116 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:08.130074 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:04:11.542616 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.542580 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c"] Jul 15 20:04:11.543052 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.543027 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4312a8-e16f-4e92-a2f8-8bc5ecf024ac" containerName="istio-proxy" Jul 15 20:04:11.546046 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.546027 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.548562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.548535 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12e2475c\"/\"openshift-service-ca.crt\"" Jul 15 20:04:11.548562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.548554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12e2475c\"/\"default-dockercfg-cj42d\"" Jul 15 20:04:11.548730 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.548538 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12e2475c\"/\"e2e-lifecycle-model-12e2475c-kserve-self-signed-certs\"" Jul 15 20:04:11.548730 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.548637 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12e2475c\"/\"kube-root-ca.crt\"" Jul 15 20:04:11.556454 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.556429 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c"] Jul 15 20:04:11.646158 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.646115 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-home\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.646158 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.646157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-tmp-dir\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.646391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.646175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-model-cache\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.646391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.646245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-dshm\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.646391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.646274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqf4c\" (UniqueName: \"kubernetes.io/projected/2b6628ed-1198-4eb8-b058-1735d020cd9b-kube-api-access-bqf4c\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.646391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.646334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6628ed-1198-4eb8-b058-1735d020cd9b-tls-certs\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.747694 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.747660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-model-cache\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.747898 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.747711 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-dshm\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.747898 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.747878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqf4c\" (UniqueName: \"kubernetes.io/projected/2b6628ed-1198-4eb8-b058-1735d020cd9b-kube-api-access-bqf4c\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.747996 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.747984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6628ed-1198-4eb8-b058-1735d020cd9b-tls-certs\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.748092 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.748077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-home\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.748182 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.748105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-tmp-dir\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.748182 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.748154 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-model-cache\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.748409 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.748384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-home\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.748547 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.748424 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-tmp-dir\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.750107 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.750083 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-dshm\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.750931 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.750908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6628ed-1198-4eb8-b058-1735d020cd9b-tls-certs\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.756388 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.756369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqf4c\" (UniqueName: \"kubernetes.io/projected/2b6628ed-1198-4eb8-b058-1735d020cd9b-kube-api-access-bqf4c\") pod \"e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:11.856417 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:11.856332 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:12.047731 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:12.047704 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c"] Jul 15 20:04:12.049764 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:04:12.049734 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6628ed_1198_4eb8_b058_1735d020cd9b.slice/crio-128265ece12f517fa894c52075b63e4849ff5b6995c7fe7d8eff0f75569d0af6 WatchSource:0}: Error finding container 128265ece12f517fa894c52075b63e4849ff5b6995c7fe7d8eff0f75569d0af6: Status 404 returned error can't find the container with id 128265ece12f517fa894c52075b63e4849ff5b6995c7fe7d8eff0f75569d0af6 Jul 15 20:04:12.765596 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:12.765553 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" event={"ID":"2b6628ed-1198-4eb8-b058-1735d020cd9b","Type":"ContainerStarted","Data":"981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5"} Jul 15 20:04:12.766081 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:12.765607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" event={"ID":"2b6628ed-1198-4eb8-b058-1735d020cd9b","Type":"ContainerStarted","Data":"128265ece12f517fa894c52075b63e4849ff5b6995c7fe7d8eff0f75569d0af6"} Jul 15 20:04:15.784951 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:15.784866 2573 generic.go:356] "Generic (PLEG): container finished" podID="54889e9d-448b-4243-89aa-760ceb26766c" containerID="d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec" exitCode=0 Jul 15 20:04:15.785388 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:15.784947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" event={"ID":"54889e9d-448b-4243-89aa-760ceb26766c","Type":"ContainerDied","Data":"d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec"} Jul 15 20:04:15.785469 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:15.785435 2573 scope.go:117] "RemoveContainer" containerID="d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec" Jul 15 20:04:15.786721 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:15.786665 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:04:15.786850 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:15.786762 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:04:15.786850 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:15.786833 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:04:15.805608 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:15.805563 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" podStartSLOduration=4.805549072 podStartE2EDuration="4.805549072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:04:12.789191727 +0000 UTC m=+2134.650396083" watchObservedRunningTime="2026-07-15 20:04:15.805549072 +0000 UTC m=+2137.666753425" Jul 15 20:04:16.709282 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:16.709247 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:16.796301 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:16.796264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" event={"ID":"54889e9d-448b-4243-89aa-760ceb26766c","Type":"ContainerStarted","Data":"57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef"} Jul 15 20:04:17.033916 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:17.033821 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:17.034989 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:17.034960 2573 patch_prober.go:28] interesting pod/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:04:17.035119 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:17.035013 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 20:04:18.034438 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:18.034396 2573 patch_prober.go:28] interesting pod/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:04:18.034897 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:18.034455 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 20:04:19.033927 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:19.033895 2573 patch_prober.go:28] interesting pod/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:04:19.034109 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:19.033956 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 20:04:20.034302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:20.034267 2573 patch_prober.go:28] interesting pod/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:04:20.034729 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:20.034323 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 20:04:21.037777 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:21.037748 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:21.038286 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:21.038163 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:21.038782 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:21.038763 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:04:21.857186 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:21.857155 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:21.857388 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:21.857207 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:21.870447 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:21.870424 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:22.040507 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:22.040471 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:04:22.842899 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:22.842868 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:04:34.433279 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:34.433248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:04:45.142302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:45.142264 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:04:54.336479 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:54.336448 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:04:58.045942 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:04:58.045890 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c"] Jul 15 20:05:13.067937 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.067894 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" podUID="2b6628ed-1198-4eb8-b058-1735d020cd9b" containerName="main" containerID="cri-o://981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5" gracePeriod=45 Jul 15 20:05:13.317680 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.317655 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:05:13.402635 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.402520 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6628ed-1198-4eb8-b058-1735d020cd9b-tls-certs\") pod \"2b6628ed-1198-4eb8-b058-1735d020cd9b\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " Jul 15 20:05:13.402635 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.402586 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-dshm\") pod \"2b6628ed-1198-4eb8-b058-1735d020cd9b\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " Jul 15 20:05:13.402635 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.402621 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-tmp-dir\") pod \"2b6628ed-1198-4eb8-b058-1735d020cd9b\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " Jul 15 20:05:13.402962 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.402699 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-model-cache\") pod \"2b6628ed-1198-4eb8-b058-1735d020cd9b\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " Jul 15 20:05:13.402962 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.402718 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqf4c\" (UniqueName: \"kubernetes.io/projected/2b6628ed-1198-4eb8-b058-1735d020cd9b-kube-api-access-bqf4c\") pod \"2b6628ed-1198-4eb8-b058-1735d020cd9b\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " Jul 15 20:05:13.402962 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.402742 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-home\") pod \"2b6628ed-1198-4eb8-b058-1735d020cd9b\" (UID: \"2b6628ed-1198-4eb8-b058-1735d020cd9b\") " Jul 15 20:05:13.403130 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.403031 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b6628ed-1198-4eb8-b058-1735d020cd9b" (UID: "2b6628ed-1198-4eb8-b058-1735d020cd9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:13.403130 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.403078 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b6628ed-1198-4eb8-b058-1735d020cd9b" (UID: "2b6628ed-1198-4eb8-b058-1735d020cd9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:13.403130 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.403107 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-home" (OuterVolumeSpecName: "home") pod "2b6628ed-1198-4eb8-b058-1735d020cd9b" (UID: "2b6628ed-1198-4eb8-b058-1735d020cd9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:13.405067 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.405045 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-dshm" (OuterVolumeSpecName: "dshm") pod "2b6628ed-1198-4eb8-b058-1735d020cd9b" (UID: "2b6628ed-1198-4eb8-b058-1735d020cd9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:13.405067 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.405054 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6628ed-1198-4eb8-b058-1735d020cd9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b6628ed-1198-4eb8-b058-1735d020cd9b" (UID: "2b6628ed-1198-4eb8-b058-1735d020cd9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:05:13.405222 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.405104 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6628ed-1198-4eb8-b058-1735d020cd9b-kube-api-access-bqf4c" (OuterVolumeSpecName: "kube-api-access-bqf4c") pod "2b6628ed-1198-4eb8-b058-1735d020cd9b" (UID: "2b6628ed-1198-4eb8-b058-1735d020cd9b"). InnerVolumeSpecName "kube-api-access-bqf4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:05:13.504302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.504246 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6628ed-1198-4eb8-b058-1735d020cd9b-tls-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:13.504302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.504297 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-dshm\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:13.504302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.504306 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-tmp-dir\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:13.504302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.504314 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-model-cache\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:13.504302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.504324 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqf4c\" (UniqueName: \"kubernetes.io/projected/2b6628ed-1198-4eb8-b058-1735d020cd9b-kube-api-access-bqf4c\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:13.504973 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:13.504333 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6628ed-1198-4eb8-b058-1735d020cd9b-home\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:14.087310 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.087272 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b6628ed-1198-4eb8-b058-1735d020cd9b" containerID="981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5" exitCode=0 Jul 15 20:05:14.087746 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.087321 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" Jul 15 20:05:14.087746 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.087357 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" event={"ID":"2b6628ed-1198-4eb8-b058-1735d020cd9b","Type":"ContainerDied","Data":"981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5"} Jul 15 20:05:14.087746 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.087411 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c" event={"ID":"2b6628ed-1198-4eb8-b058-1735d020cd9b","Type":"ContainerDied","Data":"128265ece12f517fa894c52075b63e4849ff5b6995c7fe7d8eff0f75569d0af6"} Jul 15 20:05:14.087746 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.087429 2573 scope.go:117] "RemoveContainer" containerID="981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5" Jul 15 20:05:14.096720 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.096697 2573 scope.go:117] "RemoveContainer" containerID="981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5" Jul 15 20:05:14.097019 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:05:14.096995 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5\": container with ID starting with 981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5 not found: ID does not exist" containerID="981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5" Jul 15 20:05:14.097101 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.097033 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5"} err="failed to get container status \"981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5\": rpc error: code = NotFound desc = could not find container \"981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5\": container with ID starting with 981904aea87dc86814a122872e11197dc2a4911442855a4cb5fc39ba6857a0a5 not found: ID does not exist" Jul 15 20:05:14.106668 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.106635 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c"] Jul 15 20:05:14.111818 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.111786 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-12e2475c/e2e-lifecycle-model-12e2475c-kserve-594bcf77cf-n8m5c"] Jul 15 20:05:14.695821 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:14.695783 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6628ed-1198-4eb8-b058-1735d020cd9b" path="/var/lib/kubelet/pods/2b6628ed-1198-4eb8-b058-1735d020cd9b/volumes" Jul 15 20:05:50.798816 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:50.798763 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr"] Jul 15 20:05:50.799383 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:50.799025 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" containerID="cri-o://57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef" gracePeriod=30 Jul 15 20:05:51.040689 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:51.040660 2573 patch_prober.go:28] interesting pod/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:05:51.040865 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:51.040725 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:05:54.442947 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:54.442912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:05:56.068263 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.068203 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:05:56.168218 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168187 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54889e9d-448b-4243-89aa-760ceb26766c-istiod-ca-cert\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168400 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168239 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-istio-token\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168400 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168264 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-data\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168400 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168290 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdr9m\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-kube-api-access-xdr9m\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168400 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168330 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54889e9d-448b-4243-89aa-760ceb26766c-istio-podinfo\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168400 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168364 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-credential-socket\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168400 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168396 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-socket\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168459 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-envoy\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168513 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-certs\") pod \"54889e9d-448b-4243-89aa-760ceb26766c\" (UID: \"54889e9d-448b-4243-89aa-760ceb26766c\") " Jul 15 20:05:56.168798 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168670 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54889e9d-448b-4243-89aa-760ceb26766c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:05:56.168897 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168855 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:56.168897 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168883 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:56.169009 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.168974 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:56.169374 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.169330 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-data" (OuterVolumeSpecName: "istio-data") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:56.170634 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.170609 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-istio-token" (OuterVolumeSpecName: "istio-token") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:05:56.170747 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.170722 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-kube-api-access-xdr9m" (OuterVolumeSpecName: "kube-api-access-xdr9m") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "kube-api-access-xdr9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:05:56.171321 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.171301 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54889e9d-448b-4243-89aa-760ceb26766c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:05:56.171321 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.171307 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54889e9d-448b-4243-89aa-760ceb26766c" (UID: "54889e9d-448b-4243-89aa-760ceb26766c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:56.269644 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269612 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.269644 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269641 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.269644 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269650 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdr9m\" (UniqueName: \"kubernetes.io/projected/54889e9d-448b-4243-89aa-760ceb26766c-kube-api-access-xdr9m\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.269644 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269660 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54889e9d-448b-4243-89aa-760ceb26766c-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.270014 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269669 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.270014 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269677 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.270014 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269684 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.270014 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269692 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54889e9d-448b-4243-89aa-760ceb26766c-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.270014 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.269699 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54889e9d-448b-4243-89aa-760ceb26766c-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:05:56.298111 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.298073 2573 generic.go:356] "Generic (PLEG): container finished" podID="54889e9d-448b-4243-89aa-760ceb26766c" containerID="57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef" exitCode=0 Jul 15 20:05:56.298288 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.298135 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" Jul 15 20:05:56.298288 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.298154 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" event={"ID":"54889e9d-448b-4243-89aa-760ceb26766c","Type":"ContainerDied","Data":"57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef"} Jul 15 20:05:56.298288 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.298209 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr" event={"ID":"54889e9d-448b-4243-89aa-760ceb26766c","Type":"ContainerDied","Data":"a653388565fa9f0d7280ba0d163f7a5297f4a24e38c5893e7883adfb38032bfa"} Jul 15 20:05:56.298288 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.298230 2573 scope.go:117] "RemoveContainer" containerID="57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef" Jul 15 20:05:56.308586 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.308568 2573 scope.go:117] "RemoveContainer" containerID="d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec" Jul 15 20:05:56.316990 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.316939 2573 scope.go:117] "RemoveContainer" containerID="57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef" Jul 15 20:05:56.317381 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:05:56.317350 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef\": container with ID starting with 57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef not found: ID does not exist" containerID="57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef" Jul 15 20:05:56.317504 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.317406 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef"} err="failed to get container status \"57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef\": rpc error: code = NotFound desc = could not find container \"57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef\": container with ID starting with 57dc0ec85a89fb0e6b48c9edafd6999fc0ffc64694d07536cc2d7e204393afef not found: ID does not exist" Jul 15 20:05:56.317504 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.317436 2573 scope.go:117] "RemoveContainer" containerID="d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec" Jul 15 20:05:56.317824 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:05:56.317783 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec\": container with ID starting with d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec not found: ID does not exist" containerID="d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec" Jul 15 20:05:56.317893 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.317828 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec"} err="failed to get container status \"d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec\": rpc error: code = NotFound desc = could not find container \"d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec\": container with ID starting with d3ccaa92490d6811ef270874cdadd352280b466ba1e49337ef6852dbea1802ec not found: ID does not exist" Jul 15 20:05:56.319045 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.319001 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr"] Jul 15 20:05:56.326649 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.326627 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12e2475c-openshift-default-6b57d7b68c-prgjr"] Jul 15 20:05:56.696740 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:05:56.696705 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54889e9d-448b-4243-89aa-760ceb26766c" path="/var/lib/kubelet/pods/54889e9d-448b-4243-89aa-760ceb26766c/volumes" Jul 15 20:06:07.035677 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:07.035596 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:06:46.292609 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.292573 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf"] Jul 15 20:06:46.293161 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293145 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" Jul 15 20:06:46.293208 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293165 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" Jul 15 20:06:46.293208 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293178 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" Jul 15 20:06:46.293208 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293186 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" Jul 15 20:06:46.293304 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293211 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6628ed-1198-4eb8-b058-1735d020cd9b" containerName="main" Jul 15 20:06:46.293304 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293219 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6628ed-1198-4eb8-b058-1735d020cd9b" containerName="main" Jul 15 20:06:46.293382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293304 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" Jul 15 20:06:46.293382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.293317 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6628ed-1198-4eb8-b058-1735d020cd9b" containerName="main" Jul 15 20:06:46.296596 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.296577 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.298884 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.298865 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a86716-openshift-default-dockercfg-gfjj9\"" Jul 15 20:06:46.302843 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.302793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-envoy\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.302977 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.302868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-certs\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.303053 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.302981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-socket\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.303053 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.303022 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaab058-c960-41f6-b677-d31d50825e6a-istiod-ca-cert\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.303169 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.303053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-credential-socket\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.303169 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.303094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-data\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.303169 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.303157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaab058-c960-41f6-b677-d31d50825e6a-istio-podinfo\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.303304 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.303182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwq4\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-kube-api-access-4hwq4\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.303304 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.303210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-istio-token\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.305976 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.305951 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf"] Jul 15 20:06:46.404728 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaab058-c960-41f6-b677-d31d50825e6a-istio-podinfo\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwq4\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-kube-api-access-4hwq4\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-istio-token\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-envoy\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-certs\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-socket\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaab058-c960-41f6-b677-d31d50825e6a-istiod-ca-cert\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-credential-socket\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.404993 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-data\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.405352 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-certs\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405890 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.405673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaab058-c960-41f6-b677-d31d50825e6a-istiod-ca-cert\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.405983 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.405964 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-credential-socket\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.406083 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.406064 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-data\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.406151 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.406129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-socket\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.407591 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.407568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-envoy\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.407723 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.407708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaab058-c960-41f6-b677-d31d50825e6a-istio-podinfo\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.412143 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.412116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwq4\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-kube-api-access-4hwq4\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.412430 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.412408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-istio-token\") pod \"e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.608773 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.608683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:46.748796 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.748760 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf"] Jul 15 20:06:46.750260 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:06:46.750234 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaab058_c960_41f6_b677_d31d50825e6a.slice/crio-2b67721566f2fd1db32b8261d3afc946e3e8baeb6e09b28828ff91a253c9a2a2 WatchSource:0}: Error finding container 2b67721566f2fd1db32b8261d3afc946e3e8baeb6e09b28828ff91a253c9a2a2: Status 404 returned error can't find the container with id 2b67721566f2fd1db32b8261d3afc946e3e8baeb6e09b28828ff91a253c9a2a2 Jul 15 20:06:46.752378 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.752341 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:06:46.752461 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.752410 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:06:46.752461 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:46.752438 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:06:47.565398 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:47.565357 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" event={"ID":"bfaab058-c960-41f6-b677-d31d50825e6a","Type":"ContainerStarted","Data":"9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36"} Jul 15 20:06:47.565895 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:47.565416 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" event={"ID":"bfaab058-c960-41f6-b677-d31d50825e6a","Type":"ContainerStarted","Data":"2b67721566f2fd1db32b8261d3afc946e3e8baeb6e09b28828ff91a253c9a2a2"} Jul 15 20:06:47.585639 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:47.585592 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" podStartSLOduration=1.5855771669999998 podStartE2EDuration="1.585577167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:06:47.583608731 +0000 UTC m=+2289.444813111" watchObservedRunningTime="2026-07-15 20:06:47.585577167 +0000 UTC m=+2289.446781581" Jul 15 20:06:47.609736 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:47.609709 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:47.614876 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:47.614852 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:48.572568 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:48.572529 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:48.576132 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:48.576107 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:06:54.434595 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:54.434562 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:06:56.168432 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.168395 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7"] Jul 15 20:06:56.168932 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.168835 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="54889e9d-448b-4243-89aa-760ceb26766c" containerName="istio-proxy" Jul 15 20:06:56.172140 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.172112 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.174403 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.174385 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c9860b-openshift-default-dockercfg-lq7nh\"" Jul 15 20:06:56.182862 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.182776 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7"] Jul 15 20:06:56.306302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.306261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvsv\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-kube-api-access-9jvsv\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.306605 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.306583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-socket\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.306720 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.306706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-token\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.306859 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.306838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-credential-socket\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.307100 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.307080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-podinfo\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.307239 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.307221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-envoy\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.307355 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.307341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-certs\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.307473 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.307458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-data\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.307578 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.307565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istiod-ca-cert\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.408916 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.408877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-socket\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.408916 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.408917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-token\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.408935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-credential-socket\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.408969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-podinfo\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.408990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-envoy\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-certs\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-data\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istiod-ca-cert\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409135 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvsv\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-kube-api-access-9jvsv\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409596 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409335 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-socket\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409596 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409374 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-certs\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409596 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-credential-socket\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409596 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-data\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.409883 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.409862 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istiod-ca-cert\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.411555 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.411528 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-envoy\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.411661 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.411587 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-podinfo\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.417220 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.417168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvsv\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-kube-api-access-9jvsv\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.417405 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.417386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-token\") pod \"e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.495139 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.494768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:56.661416 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.661388 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7"] Jul 15 20:06:56.663712 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:06:56.663685 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4941add7_c2b3_4f1b_a0a3_d9232c8f8dfc.slice/crio-68c7bee9d3a2682f834d150a7b250e7a30134afeb8e250c8716b0da98c10d725 WatchSource:0}: Error finding container 68c7bee9d3a2682f834d150a7b250e7a30134afeb8e250c8716b0da98c10d725: Status 404 returned error can't find the container with id 68c7bee9d3a2682f834d150a7b250e7a30134afeb8e250c8716b0da98c10d725 Jul 15 20:06:56.665751 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.665717 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:06:56.665868 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.665781 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:06:56.665868 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:56.665837 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:06:57.619329 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:57.619296 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" event={"ID":"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc","Type":"ContainerStarted","Data":"69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0"} Jul 15 20:06:57.619787 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:57.619354 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" event={"ID":"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc","Type":"ContainerStarted","Data":"68c7bee9d3a2682f834d150a7b250e7a30134afeb8e250c8716b0da98c10d725"} Jul 15 20:06:57.645135 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:57.645083 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" podStartSLOduration=1.645067149 podStartE2EDuration="1.645067149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:06:57.642283262 +0000 UTC m=+2299.503487616" watchObservedRunningTime="2026-07-15 20:06:57.645067149 +0000 UTC m=+2299.506271499" Jul 15 20:06:58.498364 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:58.498326 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:58.505422 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:58.505391 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:58.624127 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:58.624093 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:06:58.624980 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:06:58.624963 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:07:02.646721 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:02.646618 2573 generic.go:356] "Generic (PLEG): container finished" podID="bfaab058-c960-41f6-b677-d31d50825e6a" containerID="9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36" exitCode=0 Jul 15 20:07:02.646721 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:02.646699 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" event={"ID":"bfaab058-c960-41f6-b677-d31d50825e6a","Type":"ContainerDied","Data":"9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36"} Jul 15 20:07:02.647245 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:02.647105 2573 scope.go:117] "RemoveContainer" containerID="9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36" Jul 15 20:07:02.648140 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:02.648112 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:07:02.648209 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:02.648179 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:07:02.648246 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:02.648211 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:07:03.573259 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:03.573221 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:07:03.653566 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:03.653535 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" event={"ID":"bfaab058-c960-41f6-b677-d31d50825e6a","Type":"ContainerStarted","Data":"3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9"} Jul 15 20:07:04.609171 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:04.609135 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:07:04.610446 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:04.610415 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 20:07:04.610563 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:04.610485 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 20:07:05.609673 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:05.609626 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 20:07:05.610121 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:05.609691 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 20:07:06.609397 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:06.609356 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 20:07:06.609595 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:06.609425 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 20:07:07.612760 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:07.612734 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:07:07.613230 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:07.613102 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:07:07.613816 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:07.613787 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:07:10.028933 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:10.028889 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:07:18.742495 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:18.742464 2573 generic.go:356] "Generic (PLEG): container finished" podID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerID="69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0" exitCode=0 Jul 15 20:07:18.742944 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:18.742521 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" event={"ID":"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc","Type":"ContainerDied","Data":"69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0"} Jul 15 20:07:18.742944 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:18.742896 2573 scope.go:117] "RemoveContainer" containerID="69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0" Jul 15 20:07:18.744053 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:18.744021 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:07:18.744132 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:18.744098 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:07:18.744170 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:18.744138 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:07:19.749227 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:19.749191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" event={"ID":"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc","Type":"ContainerStarted","Data":"d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e"} Jul 15 20:07:20.496230 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:20.495771 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:07:20.496448 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:20.496333 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:07:20.496448 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:20.496385 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 20:07:21.495920 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.495878 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:07:21.496425 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.495943 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 20:07:21.854614 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.854524 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf"] Jul 15 20:07:21.856981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.856964 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:21.859913 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.859890 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a86716\"/\"default-dockercfg-np8m2\"" Jul 15 20:07:21.860059 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.859963 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a86716\"/\"openshift-service-ca.crt\"" Jul 15 20:07:21.860059 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.859963 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a86716\"/\"kube-root-ca.crt\"" Jul 15 20:07:21.860059 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.859962 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a86716\"/\"test-model-a86716-kserve-self-signed-certs\"" Jul 15 20:07:21.868185 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.868141 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf"] Jul 15 20:07:21.947585 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.947550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrr9\" (UniqueName: \"kubernetes.io/projected/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-kube-api-access-4zrr9\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:21.947760 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.947600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tmp-dir\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:21.947760 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.947628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-model-cache\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:21.947760 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.947694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tls-certs\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:21.947760 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.947750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-home\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:21.947939 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:21.947815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-dshm\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.048438 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.048403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-home\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.048665 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.048478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-dshm\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.048665 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.048516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrr9\" (UniqueName: \"kubernetes.io/projected/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-kube-api-access-4zrr9\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.048665 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.048560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tmp-dir\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.048665 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.048606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-model-cache\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.048665 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.048645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tls-certs\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.048992 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.048912 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-home\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.049350 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.049324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tmp-dir\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.049611 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.049589 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-model-cache\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.051535 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.051501 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-dshm\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.051760 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.051736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tls-certs\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.058125 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.058096 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrr9\" (UniqueName: \"kubernetes.io/projected/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-kube-api-access-4zrr9\") pod \"test-model-a86716-kserve-5949fcf4c5-vqdqf\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.168183 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.168153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:22.352378 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.352345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf"] Jul 15 20:07:22.354220 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:07:22.354195 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32f7b01f_4f30_4787_9e27_bbaddfc36b3e.slice/crio-cd332d306173f7512aa8734bd0fb07340f89930ae1304deedb276be16f1d7823 WatchSource:0}: Error finding container cd332d306173f7512aa8734bd0fb07340f89930ae1304deedb276be16f1d7823: Status 404 returned error can't find the container with id cd332d306173f7512aa8734bd0fb07340f89930ae1304deedb276be16f1d7823 Jul 15 20:07:22.496291 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.496222 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:07:22.496842 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.496294 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 20:07:22.765495 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.765394 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" event={"ID":"32f7b01f-4f30-4787-9e27-bbaddfc36b3e","Type":"ContainerStarted","Data":"eedd61ac812b715523840ef174e98698b31762455564607026913631694bdb6f"} Jul 15 20:07:22.765495 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.765438 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" event={"ID":"32f7b01f-4f30-4787-9e27-bbaddfc36b3e","Type":"ContainerStarted","Data":"cd332d306173f7512aa8734bd0fb07340f89930ae1304deedb276be16f1d7823"} Jul 15 20:07:22.785164 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:22.785099 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" podStartSLOduration=1.7850787769999998 podStartE2EDuration="1.785078777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:22.781768101 +0000 UTC m=+2324.642972454" watchObservedRunningTime="2026-07-15 20:07:22.785078777 +0000 UTC m=+2324.646283131" Jul 15 20:07:23.496119 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:23.496079 2573 patch_prober.go:28] interesting pod/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:07:23.496309 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:23.496218 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 20:07:24.499644 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:24.499613 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:07:24.500051 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:24.500016 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:07:24.500667 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:24.500648 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:07:25.536579 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:25.536540 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:07:32.168826 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:32.168715 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:32.168826 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:32.168786 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:32.181846 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:32.181793 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:32.828743 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:32.828713 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:07:36.329846 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:36.329800 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:07:52.131389 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:07:52.131349 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:08:06.837544 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:06.837510 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:08:13.933115 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:13.933083 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf"] Jul 15 20:08:19.248095 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:19.248059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:08:28.954190 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:28.954145 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" podUID="32f7b01f-4f30-4787-9e27-bbaddfc36b3e" containerName="main" containerID="cri-o://eedd61ac812b715523840ef174e98698b31762455564607026913631694bdb6f" gracePeriod=45 Jul 15 20:08:29.115121 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.115081 2573 generic.go:356] "Generic (PLEG): container finished" podID="32f7b01f-4f30-4787-9e27-bbaddfc36b3e" containerID="eedd61ac812b715523840ef174e98698b31762455564607026913631694bdb6f" exitCode=0 Jul 15 20:08:29.115321 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.115190 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" event={"ID":"32f7b01f-4f30-4787-9e27-bbaddfc36b3e","Type":"ContainerDied","Data":"eedd61ac812b715523840ef174e98698b31762455564607026913631694bdb6f"} Jul 15 20:08:29.136576 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.136540 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:08:29.221564 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.221537 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:08:29.282239 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282202 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zrr9\" (UniqueName: \"kubernetes.io/projected/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-kube-api-access-4zrr9\") pod \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " Jul 15 20:08:29.282422 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282259 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tmp-dir\") pod \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " Jul 15 20:08:29.282422 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282292 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-model-cache\") pod \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " Jul 15 20:08:29.282422 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282322 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-dshm\") pod \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " Jul 15 20:08:29.282422 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282394 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-home\") pod \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " Jul 15 20:08:29.282645 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282424 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tls-certs\") pod \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\" (UID: \"32f7b01f-4f30-4787-9e27-bbaddfc36b3e\") " Jul 15 20:08:29.282645 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282562 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32f7b01f-4f30-4787-9e27-bbaddfc36b3e" (UID: "32f7b01f-4f30-4787-9e27-bbaddfc36b3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:29.282645 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282587 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "32f7b01f-4f30-4787-9e27-bbaddfc36b3e" (UID: "32f7b01f-4f30-4787-9e27-bbaddfc36b3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:29.282799 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282778 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tmp-dir\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:29.282799 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282797 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-model-cache\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:29.282913 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.282837 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-home" (OuterVolumeSpecName: "home") pod "32f7b01f-4f30-4787-9e27-bbaddfc36b3e" (UID: "32f7b01f-4f30-4787-9e27-bbaddfc36b3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:29.284631 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.284603 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-kube-api-access-4zrr9" (OuterVolumeSpecName: "kube-api-access-4zrr9") pod "32f7b01f-4f30-4787-9e27-bbaddfc36b3e" (UID: "32f7b01f-4f30-4787-9e27-bbaddfc36b3e"). InnerVolumeSpecName "kube-api-access-4zrr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:29.284786 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.284771 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32f7b01f-4f30-4787-9e27-bbaddfc36b3e" (UID: "32f7b01f-4f30-4787-9e27-bbaddfc36b3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:08:29.284786 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.284778 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-dshm" (OuterVolumeSpecName: "dshm") pod "32f7b01f-4f30-4787-9e27-bbaddfc36b3e" (UID: "32f7b01f-4f30-4787-9e27-bbaddfc36b3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:29.383776 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.383739 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zrr9\" (UniqueName: \"kubernetes.io/projected/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-kube-api-access-4zrr9\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:29.383776 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.383767 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-dshm\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:29.383776 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.383777 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-home\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:29.383776 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:29.383784 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7b01f-4f30-4787-9e27-bbaddfc36b3e-tls-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:30.121058 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:30.121022 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" event={"ID":"32f7b01f-4f30-4787-9e27-bbaddfc36b3e","Type":"ContainerDied","Data":"cd332d306173f7512aa8734bd0fb07340f89930ae1304deedb276be16f1d7823"} Jul 15 20:08:30.121526 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:30.121069 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf" Jul 15 20:08:30.121526 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:30.121085 2573 scope.go:117] "RemoveContainer" containerID="eedd61ac812b715523840ef174e98698b31762455564607026913631694bdb6f" Jul 15 20:08:30.141641 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:30.141612 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf"] Jul 15 20:08:30.145377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:30.145351 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a86716/test-model-a86716-kserve-5949fcf4c5-vqdqf"] Jul 15 20:08:30.697246 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:30.697209 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f7b01f-4f30-4787-9e27-bbaddfc36b3e" path="/var/lib/kubelet/pods/32f7b01f-4f30-4787-9e27-bbaddfc36b3e/volumes" Jul 15 20:08:36.647623 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:36.647586 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf"] Jul 15 20:08:36.648193 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:36.647908 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" containerID="cri-o://3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9" gracePeriod=30 Jul 15 20:08:37.615874 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:37.615838 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:08:37.616076 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:37.615911 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:08:38.779829 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:38.778422 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 20:08:38.791058 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:38.791031 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 20:08:38.791911 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:38.791883 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 20:08:38.797832 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:38.797796 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 20:08:41.905575 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:41.905552 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:08:42.005136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005049 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-socket\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005089 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-istio-token\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005109 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-credential-socket\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005129 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaab058-c960-41f6-b677-d31d50825e6a-istiod-ca-cert\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005467 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005149 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hwq4\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-kube-api-access-4hwq4\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005467 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005280 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaab058-c960-41f6-b677-d31d50825e6a-istio-podinfo\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005467 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005318 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-certs\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005467 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005335 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-data\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005467 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005349 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:42.005467 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005384 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-envoy\") pod \"bfaab058-c960-41f6-b677-d31d50825e6a\" (UID: \"bfaab058-c960-41f6-b677-d31d50825e6a\") " Jul 15 20:08:42.005467 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005459 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:42.005901 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005619 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfaab058-c960-41f6-b677-d31d50825e6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:08:42.005901 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005625 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:42.005901 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005720 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.005901 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005742 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.005901 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005756 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.005901 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.005773 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaab058-c960-41f6-b677-d31d50825e6a-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.006512 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.006486 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:42.007576 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.007548 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfaab058-c960-41f6-b677-d31d50825e6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:08:42.007671 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.007645 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:08:42.007712 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.007665 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:42.007750 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.007727 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-kube-api-access-4hwq4" (OuterVolumeSpecName: "kube-api-access-4hwq4") pod "bfaab058-c960-41f6-b677-d31d50825e6a" (UID: "bfaab058-c960-41f6-b677-d31d50825e6a"). InnerVolumeSpecName "kube-api-access-4hwq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:42.107212 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.107173 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.107212 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.107204 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaab058-c960-41f6-b677-d31d50825e6a-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.107212 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.107212 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.107212 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.107221 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwq4\" (UniqueName: \"kubernetes.io/projected/bfaab058-c960-41f6-b677-d31d50825e6a-kube-api-access-4hwq4\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.107496 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.107230 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaab058-c960-41f6-b677-d31d50825e6a-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:08:42.185412 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.185377 2573 generic.go:356] "Generic (PLEG): container finished" podID="bfaab058-c960-41f6-b677-d31d50825e6a" containerID="3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9" exitCode=0 Jul 15 20:08:42.185604 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.185426 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" Jul 15 20:08:42.185604 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.185425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" event={"ID":"bfaab058-c960-41f6-b677-d31d50825e6a","Type":"ContainerDied","Data":"3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9"} Jul 15 20:08:42.185604 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.185556 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf" event={"ID":"bfaab058-c960-41f6-b677-d31d50825e6a","Type":"ContainerDied","Data":"2b67721566f2fd1db32b8261d3afc946e3e8baeb6e09b28828ff91a253c9a2a2"} Jul 15 20:08:42.185604 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.185584 2573 scope.go:117] "RemoveContainer" containerID="3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9" Jul 15 20:08:42.194873 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.194850 2573 scope.go:117] "RemoveContainer" containerID="9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36" Jul 15 20:08:42.203570 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.203548 2573 scope.go:117] "RemoveContainer" containerID="3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9" Jul 15 20:08:42.203979 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:08:42.203940 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9\": container with ID starting with 3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9 not found: ID does not exist" containerID="3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9" Jul 15 20:08:42.204063 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.203987 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9"} err="failed to get container status \"3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9\": rpc error: code = NotFound desc = could not find container \"3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9\": container with ID starting with 3b140cfa6783bbc0c6d567c269ecc487ef32b889615382b41e48c70e1dc6dbf9 not found: ID does not exist" Jul 15 20:08:42.204063 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.204004 2573 scope.go:117] "RemoveContainer" containerID="9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36" Jul 15 20:08:42.204273 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:08:42.204258 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36\": container with ID starting with 9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36 not found: ID does not exist" containerID="9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36" Jul 15 20:08:42.204313 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.204301 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36"} err="failed to get container status \"9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36\": rpc error: code = NotFound desc = could not find container \"9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36\": container with ID starting with 9ef40c33effa8473c0ec1a5399569e48e24ef0ea180132eabed84f247be5ec36 not found: ID does not exist" Jul 15 20:08:42.205193 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.205175 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf"] Jul 15 20:08:42.209514 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.209492 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a86716-openshift-default-c9bf77b95-6n2vf"] Jul 15 20:08:42.695371 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:42.695339 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" path="/var/lib/kubelet/pods/bfaab058-c960-41f6-b677-d31d50825e6a/volumes" Jul 15 20:08:50.836693 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:50.836659 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:08:58.655008 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:58.654975 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7"] Jul 15 20:08:58.655503 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:58.655198 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" containerID="cri-o://d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e" gracePeriod=30 Jul 15 20:08:59.599968 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.599931 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf"] Jul 15 20:08:59.600309 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.600296 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" Jul 15 20:08:59.600378 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.600310 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" Jul 15 20:08:59.600378 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.600325 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f7b01f-4f30-4787-9e27-bbaddfc36b3e" containerName="main" Jul 15 20:08:59.600378 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.600330 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f7b01f-4f30-4787-9e27-bbaddfc36b3e" containerName="main" Jul 15 20:08:59.600536 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.600387 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f7b01f-4f30-4787-9e27-bbaddfc36b3e" containerName="main" Jul 15 20:08:59.600536 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.600398 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" Jul 15 20:08:59.600536 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.600405 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" Jul 15 20:08:59.603581 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.603553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.605833 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.605797 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fc7b22-openshift-default-dockercfg-qll5v\"" Jul 15 20:08:59.615649 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.615625 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf"] Jul 15 20:08:59.781967 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.781863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-credential-socket\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.781967 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.781905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd65afe9-5024-4256-af97-06f4a99de069-istio-podinfo\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.781967 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.781943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlxr\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-kube-api-access-pdlxr\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.782572 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.782149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-certs\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.782572 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.782195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-istio-token\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.782572 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.782274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd65afe9-5024-4256-af97-06f4a99de069-istiod-ca-cert\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.782572 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.782319 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-envoy\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.782572 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.782335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-data\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.782572 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.782380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-socket\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883284 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-socket\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883284 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-credential-socket\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883284 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd65afe9-5024-4256-af97-06f4a99de069-istio-podinfo\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883284 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlxr\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-kube-api-access-pdlxr\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883614 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-certs\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883614 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-istio-token\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883614 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd65afe9-5024-4256-af97-06f4a99de069-istiod-ca-cert\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883614 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-envoy\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883614 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-data\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883874 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-socket\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883874 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-credential-socket\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.883874 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.883778 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-data\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.884064 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.884045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-certs\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.884360 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.884336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd65afe9-5024-4256-af97-06f4a99de069-istiod-ca-cert\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.886120 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.886097 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-envoy\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.886652 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.886632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd65afe9-5024-4256-af97-06f4a99de069-istio-podinfo\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.892929 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.892903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-istio-token\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.893133 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.893116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlxr\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-kube-api-access-pdlxr\") pod \"e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:08:59.914121 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:08:59.914093 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:00.057546 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.057516 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf"] Jul 15 20:09:00.059464 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:09:00.059432 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd65afe9_5024_4256_af97_06f4a99de069.slice/crio-94837d67dc65a9ae9adc614c9181123d6cb0b7cbc61946cf3b72f7d59fe66c9c WatchSource:0}: Error finding container 94837d67dc65a9ae9adc614c9181123d6cb0b7cbc61946cf3b72f7d59fe66c9c: Status 404 returned error can't find the container with id 94837d67dc65a9ae9adc614c9181123d6cb0b7cbc61946cf3b72f7d59fe66c9c Jul 15 20:09:00.061667 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.061624 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:09:00.061749 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.061726 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:09:00.061789 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.061772 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:09:00.276342 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.276306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" event={"ID":"fd65afe9-5024-4256-af97-06f4a99de069","Type":"ContainerStarted","Data":"94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6"} Jul 15 20:09:00.276530 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.276371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" event={"ID":"fd65afe9-5024-4256-af97-06f4a99de069","Type":"ContainerStarted","Data":"94837d67dc65a9ae9adc614c9181123d6cb0b7cbc61946cf3b72f7d59fe66c9c"} Jul 15 20:09:00.298564 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.298514 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podStartSLOduration=1.298498059 podStartE2EDuration="1.298498059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:09:00.297644185 +0000 UTC m=+2422.158848561" watchObservedRunningTime="2026-07-15 20:09:00.298498059 +0000 UTC m=+2422.159702416" Jul 15 20:09:00.914505 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:00.914472 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:01.919333 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:01.919304 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:02.287754 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:02.287654 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:02.288797 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:02.288779 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:03.918192 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:03.918129 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:09:04.023124 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023095 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-socket\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023323 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023148 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jvsv\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-kube-api-access-9jvsv\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023323 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023183 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-token\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023323 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023202 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-certs\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023323 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023256 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-credential-socket\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023323 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023307 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istiod-ca-cert\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023331 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-envoy\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023357 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-data\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023399 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-podinfo\") pod \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\" (UID: \"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc\") " Jul 15 20:09:04.023620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023547 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:04.023898 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023653 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:04.023898 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023719 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:04.023898 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023741 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:09:04.023898 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023858 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.023898 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.023879 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.024603 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.024576 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-data" (OuterVolumeSpecName: "istio-data") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:04.025535 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.025509 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-token" (OuterVolumeSpecName: "istio-token") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:09:04.025743 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.025723 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:09:04.025841 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.025739 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-kube-api-access-9jvsv" (OuterVolumeSpecName: "kube-api-access-9jvsv") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "kube-api-access-9jvsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:09:04.025841 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.025719 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" (UID: "4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:04.125153 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.125115 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvsv\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-kube-api-access-9jvsv\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.125153 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.125145 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.125153 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.125161 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.125392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.125172 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.125392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.125182 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.125392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.125192 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.125392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.125204 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:04.299503 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.299398 2573 generic.go:356] "Generic (PLEG): container finished" podID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerID="d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e" exitCode=0 Jul 15 20:09:04.299503 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.299459 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" Jul 15 20:09:04.299503 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.299484 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" event={"ID":"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc","Type":"ContainerDied","Data":"d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e"} Jul 15 20:09:04.299790 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.299538 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7" event={"ID":"4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc","Type":"ContainerDied","Data":"68c7bee9d3a2682f834d150a7b250e7a30134afeb8e250c8716b0da98c10d725"} Jul 15 20:09:04.299790 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.299561 2573 scope.go:117] "RemoveContainer" containerID="d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e" Jul 15 20:09:04.309412 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.309395 2573 scope.go:117] "RemoveContainer" containerID="69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0" Jul 15 20:09:04.317198 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.317176 2573 scope.go:117] "RemoveContainer" containerID="d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e" Jul 15 20:09:04.317443 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:09:04.317424 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e\": container with ID starting with d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e not found: ID does not exist" containerID="d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e" Jul 15 20:09:04.317531 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.317454 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e"} err="failed to get container status \"d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e\": rpc error: code = NotFound desc = could not find container \"d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e\": container with ID starting with d47520716547c0ba0bcb264631011d0307f188b5499c62a4861a8f3705d1e59e not found: ID does not exist" Jul 15 20:09:04.317531 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.317481 2573 scope.go:117] "RemoveContainer" containerID="69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0" Jul 15 20:09:04.317754 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:09:04.317738 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0\": container with ID starting with 69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0 not found: ID does not exist" containerID="69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0" Jul 15 20:09:04.317821 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.317759 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0"} err="failed to get container status \"69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0\": rpc error: code = NotFound desc = could not find container \"69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0\": container with ID starting with 69b4a01696058017e882c1d46d430f000e708c90d4dad8fb5c6acb02e4ee3cb0 not found: ID does not exist" Jul 15 20:09:04.320981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.320959 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7"] Jul 15 20:09:04.324594 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.324575 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c9860b-openshift-default-6b89759c7f-pgmk7"] Jul 15 20:09:04.696333 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:04.696291 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" path="/var/lib/kubelet/pods/4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc/volumes" Jul 15 20:09:10.343242 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:10.343211 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:09:17.972303 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:17.971274 2573 patch_prober.go:28] interesting pod/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:36212->10.133.0.54:15021: read: connection reset by peer" start-of-body= Jul 15 20:09:17.972303 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:17.971357 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:36212->10.133.0.54:15021: read: connection reset by peer" Jul 15 20:09:18.372846 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:18.372731 2573 generic.go:356] "Generic (PLEG): container finished" podID="fd65afe9-5024-4256-af97-06f4a99de069" containerID="94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6" exitCode=0 Jul 15 20:09:18.372846 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:18.372796 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" event={"ID":"fd65afe9-5024-4256-af97-06f4a99de069","Type":"ContainerDied","Data":"94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6"} Jul 15 20:09:18.373248 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:18.373224 2573 scope.go:117] "RemoveContainer" containerID="94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6" Jul 15 20:09:18.374365 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:18.374332 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:09:18.374461 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:18.374413 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:09:18.374525 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:18.374475 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:09:19.382784 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:19.382720 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" event={"ID":"fd65afe9-5024-4256-af97-06f4a99de069","Type":"ContainerStarted","Data":"efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1"} Jul 15 20:09:19.914756 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:19.914725 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:19.915003 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:19.914980 2573 patch_prober.go:28] interesting pod/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:09:19.915055 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:19.915031 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 20:09:20.915165 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:20.915130 2573 patch_prober.go:28] interesting pod/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:09:20.915578 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:20.915200 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 20:09:21.914980 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:21.914950 2573 patch_prober.go:28] interesting pod/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:09:21.915229 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:21.915004 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 20:09:22.914824 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:22.914770 2573 patch_prober.go:28] interesting pod/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:09:22.915015 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:22.914852 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 20:09:23.919062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:23.919032 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:23.919565 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:23.919446 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:23.920180 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:23.920164 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:25.346144 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:25.346110 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:09:44.635163 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:44.635133 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:09:52.219991 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:52.219957 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf"] Jul 15 20:09:52.220499 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:52.220187 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" containerID="cri-o://efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1" gracePeriod=30 Jul 15 20:09:53.922047 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:53.922009 2573 patch_prober.go:28] interesting pod/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:09:53.922529 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:53.922077 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:09:57.475362 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.475327 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:57.522154 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522072 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-data\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522154 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522104 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-certs\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522154 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522135 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd65afe9-5024-4256-af97-06f4a99de069-istio-podinfo\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522154 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522155 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-credential-socket\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522492 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522343 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-envoy\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522492 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522389 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:57.522492 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522427 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-socket\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522492 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522442 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:57.522492 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522454 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdlxr\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-kube-api-access-pdlxr\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522492 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522471 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-istio-token\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522839 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522502 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd65afe9-5024-4256-af97-06f4a99de069-istiod-ca-cert\") pod \"fd65afe9-5024-4256-af97-06f4a99de069\" (UID: \"fd65afe9-5024-4256-af97-06f4a99de069\") " Jul 15 20:09:57.522839 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522621 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:57.523132 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522836 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.523132 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522860 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.523132 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522877 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.523132 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522985 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd65afe9-5024-4256-af97-06f4a99de069-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:09:57.523132 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.522996 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:57.524582 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.524551 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd65afe9-5024-4256-af97-06f4a99de069-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:09:57.524758 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.524739 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-kube-api-access-pdlxr" (OuterVolumeSpecName: "kube-api-access-pdlxr") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "kube-api-access-pdlxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:09:57.524831 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.524756 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:09:57.524831 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.524770 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd65afe9-5024-4256-af97-06f4a99de069" (UID: "fd65afe9-5024-4256-af97-06f4a99de069"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:09:57.581459 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.581422 2573 generic.go:356] "Generic (PLEG): container finished" podID="fd65afe9-5024-4256-af97-06f4a99de069" containerID="efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1" exitCode=0 Jul 15 20:09:57.581638 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.581478 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" Jul 15 20:09:57.581638 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.581498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" event={"ID":"fd65afe9-5024-4256-af97-06f4a99de069","Type":"ContainerDied","Data":"efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1"} Jul 15 20:09:57.581638 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.581571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf" event={"ID":"fd65afe9-5024-4256-af97-06f4a99de069","Type":"ContainerDied","Data":"94837d67dc65a9ae9adc614c9181123d6cb0b7cbc61946cf3b72f7d59fe66c9c"} Jul 15 20:09:57.581638 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.581593 2573 scope.go:117] "RemoveContainer" containerID="efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1" Jul 15 20:09:57.590843 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.590824 2573 scope.go:117] "RemoveContainer" containerID="94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6" Jul 15 20:09:57.599382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.599355 2573 scope.go:117] "RemoveContainer" containerID="efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1" Jul 15 20:09:57.599708 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:09:57.599686 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1\": container with ID starting with efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1 not found: ID does not exist" containerID="efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1" Jul 15 20:09:57.599917 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.599716 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1"} err="failed to get container status \"efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1\": rpc error: code = NotFound desc = could not find container \"efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1\": container with ID starting with efec6f34c359d541038d34b260aa0ca69d7c06364251b1dd30bc2c949954ffa1 not found: ID does not exist" Jul 15 20:09:57.599917 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.599736 2573 scope.go:117] "RemoveContainer" containerID="94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6" Jul 15 20:09:57.600051 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:09:57.599996 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6\": container with ID starting with 94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6 not found: ID does not exist" containerID="94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6" Jul 15 20:09:57.600051 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.600017 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6"} err="failed to get container status \"94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6\": rpc error: code = NotFound desc = could not find container \"94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6\": container with ID starting with 94a2d16ed590607e83fa5babfed1cdaf74f9d7fcb724b5a1a03f057ec39707a6 not found: ID does not exist" Jul 15 20:09:57.601452 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.601430 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf"] Jul 15 20:09:57.603327 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.603307 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fc7b22-openshift-default-774fb9c77f-d65zf"] Jul 15 20:09:57.624080 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.624056 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlxr\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-kube-api-access-pdlxr\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.624080 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.624078 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd65afe9-5024-4256-af97-06f4a99de069-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.624207 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.624088 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd65afe9-5024-4256-af97-06f4a99de069-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.624207 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.624096 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.624207 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.624104 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd65afe9-5024-4256-af97-06f4a99de069-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:57.624207 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:57.624111 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd65afe9-5024-4256-af97-06f4a99de069-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:09:58.697147 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:09:58.697110 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd65afe9-5024-4256-af97-06f4a99de069" path="/var/lib/kubelet/pods/fd65afe9-5024-4256-af97-06f4a99de069/volumes" Jul 15 20:10:12.639823 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:12.639766 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:10:19.895276 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895240 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n"] Jul 15 20:10:19.895688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895643 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" Jul 15 20:10:19.895688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895656 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaab058-c960-41f6-b677-d31d50825e6a" containerName="istio-proxy" Jul 15 20:10:19.895688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895666 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" Jul 15 20:10:19.895688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895674 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" Jul 15 20:10:19.895688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895683 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" Jul 15 20:10:19.895688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895689 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" Jul 15 20:10:19.895905 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895710 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" Jul 15 20:10:19.895905 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895715 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" Jul 15 20:10:19.895905 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895781 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" Jul 15 20:10:19.895905 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895789 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" Jul 15 20:10:19.895905 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.895795 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" Jul 15 20:10:19.898715 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.898692 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:19.901997 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.901980 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-826a0f-openshift-default-dockercfg-qs8lg\"" Jul 15 20:10:19.913099 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:19.913077 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n"] Jul 15 20:10:20.024893 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.024855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szp7m\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-kube-api-access-szp7m\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025083 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.024912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-socket\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025083 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.024968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-data\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025083 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.024984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-token\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025083 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.025005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istiod-ca-cert\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025083 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.025030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-podinfo\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025083 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.025058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-credential-socket\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025307 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.025092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-certs\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.025307 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.025109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-envoy\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126118 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-socket\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-data\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-token\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istiod-ca-cert\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-podinfo\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-credential-socket\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-certs\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-envoy\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126315 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szp7m\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-kube-api-access-szp7m\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126740 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126565 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-socket\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126740 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-credential-socket\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126740 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126716 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-certs\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126952 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istiod-ca-cert\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.126952 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.126940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-data\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.128695 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.128673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-envoy\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.128852 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.128832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-podinfo\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.133719 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.133694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-token\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.133922 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.133903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szp7m\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-kube-api-access-szp7m\") pod \"e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.209024 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.208991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:20.355305 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.355270 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n"] Jul 15 20:10:20.356693 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:10:20.356661 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a5cf3d_0db5_4d51_b76c_5fd759a5e647.slice/crio-304eac972bcaeb897ed89fe858448b546815df40c48484c05d6868b58852deed WatchSource:0}: Error finding container 304eac972bcaeb897ed89fe858448b546815df40c48484c05d6868b58852deed: Status 404 returned error can't find the container with id 304eac972bcaeb897ed89fe858448b546815df40c48484c05d6868b58852deed Jul 15 20:10:20.358646 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.358614 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:20.358722 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.358684 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:20.358722 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.358713 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:20.697952 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.697920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" event={"ID":"40a5cf3d-0db5-4d51-b76c-5fd759a5e647","Type":"ContainerStarted","Data":"cf43d20d7d868e657e73b82ce01dac1cea69ad7bee6ed09249376913c13b46b6"} Jul 15 20:10:20.698121 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.697969 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" event={"ID":"40a5cf3d-0db5-4d51-b76c-5fd759a5e647","Type":"ContainerStarted","Data":"304eac972bcaeb897ed89fe858448b546815df40c48484c05d6868b58852deed"} Jul 15 20:10:20.718018 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:20.717961 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" podStartSLOduration=1.717939972 podStartE2EDuration="1.717939972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:10:20.715437342 +0000 UTC m=+2502.576641714" watchObservedRunningTime="2026-07-15 20:10:20.717939972 +0000 UTC m=+2502.579144326" Jul 15 20:10:21.209907 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:21.209870 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:21.214748 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:21.214723 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:21.702668 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:21.702641 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:21.703798 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:21.703777 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:27.980461 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:27.980426 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl"] Jul 15 20:10:27.980983 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:27.980898 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" Jul 15 20:10:27.980983 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:27.980916 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd65afe9-5024-4256-af97-06f4a99de069" containerName="istio-proxy" Jul 15 20:10:27.981060 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:27.980984 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4941add7-c2b3-4f1b-a0a3-d9232c8f8dfc" containerName="istio-proxy" Jul 15 20:10:27.984217 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:27.984199 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:27.986577 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:27.986554 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b72424-openshift-default-dockercfg-f467b\"" Jul 15 20:10:27.993711 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:27.993685 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl"] Jul 15 20:10:28.101699 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-socket\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.101883 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-podinfo\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.101883 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istiod-ca-cert\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.101883 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-certs\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.101883 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-data\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.101883 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-envoy\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.102097 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-token\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.102097 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-credential-socket\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.102097 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.101933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssgcp\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-kube-api-access-ssgcp\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202692 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-podinfo\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202692 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istiod-ca-cert\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202975 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-certs\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202975 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-data\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202975 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-envoy\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202975 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-token\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202975 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-credential-socket\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.202975 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssgcp\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-kube-api-access-ssgcp\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.203242 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.202989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-socket\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.203353 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.203327 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-credential-socket\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.203433 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.203394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-data\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.203482 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.203429 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-socket\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.203482 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.203455 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-certs\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.203550 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.203530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istiod-ca-cert\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.205414 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.205387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-envoy\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.205517 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.205425 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-podinfo\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.209991 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.209968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-token\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.210236 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.210219 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssgcp\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-kube-api-access-ssgcp\") pod \"e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.296888 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.296775 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:28.446572 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.446436 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl"] Jul 15 20:10:28.449287 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:10:28.449253 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764dcf5b_53b7_440a_8d68_a9d421ab31e3.slice/crio-aa263302303ff801b463c03a120bd01572a93e504ab4a8238243463d4cf22fce WatchSource:0}: Error finding container aa263302303ff801b463c03a120bd01572a93e504ab4a8238243463d4cf22fce: Status 404 returned error can't find the container with id aa263302303ff801b463c03a120bd01572a93e504ab4a8238243463d4cf22fce Jul 15 20:10:28.451612 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.451578 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:28.451717 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.451646 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:28.451717 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.451676 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:28.735930 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.735889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" event={"ID":"764dcf5b-53b7-440a-8d68-a9d421ab31e3","Type":"ContainerStarted","Data":"5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253"} Jul 15 20:10:28.736118 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.735955 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" event={"ID":"764dcf5b-53b7-440a-8d68-a9d421ab31e3","Type":"ContainerStarted","Data":"aa263302303ff801b463c03a120bd01572a93e504ab4a8238243463d4cf22fce"} Jul 15 20:10:28.757144 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.757087 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" podStartSLOduration=1.7570672 podStartE2EDuration="1.7570672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:10:28.754981334 +0000 UTC m=+2510.616185710" watchObservedRunningTime="2026-07-15 20:10:28.7570672 +0000 UTC m=+2510.618271556" Jul 15 20:10:28.836120 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:28.836084 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:10:29.297981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:29.297946 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:29.303365 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:29.303343 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:29.742340 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:29.742302 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:29.743419 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:29.743390 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:37.051732 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.051273 2573 patch_prober.go:28] interesting pod/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:57730->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 15 20:10:37.051732 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.051334 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:57730->10.133.0.55:15021: read: connection reset by peer" Jul 15 20:10:37.787768 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.786444 2573 generic.go:356] "Generic (PLEG): container finished" podID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerID="cf43d20d7d868e657e73b82ce01dac1cea69ad7bee6ed09249376913c13b46b6" exitCode=0 Jul 15 20:10:37.787768 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.786587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" event={"ID":"40a5cf3d-0db5-4d51-b76c-5fd759a5e647","Type":"ContainerDied","Data":"cf43d20d7d868e657e73b82ce01dac1cea69ad7bee6ed09249376913c13b46b6"} Jul 15 20:10:37.787768 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.787128 2573 scope.go:117] "RemoveContainer" containerID="cf43d20d7d868e657e73b82ce01dac1cea69ad7bee6ed09249376913c13b46b6" Jul 15 20:10:37.788700 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.788531 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:37.788700 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.788620 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:37.788700 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:37.788667 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:38.798120 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:38.798078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" event={"ID":"40a5cf3d-0db5-4d51-b76c-5fd759a5e647","Type":"ContainerStarted","Data":"297bfc7eed3869885c1931b694338d51b1d8a8bfe4ab1616bbbf403e1908081d"} Jul 15 20:10:39.210068 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:39.210025 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:39.210464 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:39.210439 2573 patch_prober.go:28] interesting pod/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:10:39.210562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:39.210508 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 20:10:40.210591 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:40.210120 2573 patch_prober.go:28] interesting pod/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:10:40.210591 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:40.210186 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 20:10:41.209994 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:41.209959 2573 patch_prober.go:28] interesting pod/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:10:41.210179 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:41.210023 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 20:10:42.209822 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:42.209775 2573 patch_prober.go:28] interesting pod/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 20:10:42.210227 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:42.209862 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 20:10:43.212988 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:43.212959 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:43.213504 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:43.213348 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:43.213970 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:43.213951 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:10:44.744286 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.744249 2573 patch_prober.go:28] interesting pod/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 15 20:10:44.744723 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.744319 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 15 20:10:44.835923 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.835892 2573 generic.go:356] "Generic (PLEG): container finished" podID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerID="5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253" exitCode=0 Jul 15 20:10:44.836098 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.835972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" event={"ID":"764dcf5b-53b7-440a-8d68-a9d421ab31e3","Type":"ContainerDied","Data":"5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253"} Jul 15 20:10:44.836468 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.836449 2573 scope.go:117] "RemoveContainer" containerID="5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253" Jul 15 20:10:44.837388 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.837361 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:44.837448 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.837427 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:44.837485 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:44.837459 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 20:10:45.235972 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:45.235934 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:10:45.843865 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:45.843835 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" event={"ID":"764dcf5b-53b7-440a-8d68-a9d421ab31e3","Type":"ContainerStarted","Data":"1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919"} Jul 15 20:10:46.298170 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:46.298133 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:46.298573 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:46.298473 2573 patch_prober.go:28] interesting pod/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 15 20:10:46.298745 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:46.298722 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 15 20:10:47.298682 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:47.298237 2573 patch_prober.go:28] interesting pod/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 15 20:10:47.298682 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:47.298301 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 15 20:10:48.297652 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:48.297612 2573 patch_prober.go:28] interesting pod/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 15 20:10:48.297979 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:48.297949 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 15 20:10:49.298175 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:49.298138 2573 patch_prober.go:28] interesting pod/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 15 20:10:49.298759 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:49.298730 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 15 20:10:50.302036 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:50.301997 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:50.302515 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:50.302399 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:50.303091 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:50.303071 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:10:52.039554 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.039525 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc"] Jul 15 20:10:52.042092 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.042076 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.044212 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.044196 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"kube-root-ca.crt\"" Jul 15 20:10:52.044915 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.044898 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"default-dockercfg-ccrm8\"" Jul 15 20:10:52.045016 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.044911 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"auth-test-model-826a0f-kserve-self-signed-certs\"" Jul 15 20:10:52.045016 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.044926 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"openshift-service-ca.crt\"" Jul 15 20:10:52.050717 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.050695 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc"] Jul 15 20:10:52.152548 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.152516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrwh\" (UniqueName: \"kubernetes.io/projected/db19012e-1921-4df6-8aef-591403eccdd5-kube-api-access-nfrwh\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.152742 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.152574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-dshm\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.152742 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.152637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-tmp-dir\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.152742 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.152666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-model-cache\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.152742 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.152698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db19012e-1921-4df6-8aef-591403eccdd5-tls-certs\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.152939 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.152749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-home\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.254202 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrwh\" (UniqueName: \"kubernetes.io/projected/db19012e-1921-4df6-8aef-591403eccdd5-kube-api-access-nfrwh\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.254413 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-dshm\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.254413 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-tmp-dir\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.254413 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-model-cache\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.254594 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db19012e-1921-4df6-8aef-591403eccdd5-tls-certs\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.254594 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-home\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.254729 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254699 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-tmp-dir\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.255055 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254760 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-model-cache\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.255055 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.254930 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-home\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.256607 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.256586 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-dshm\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.257171 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.257152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db19012e-1921-4df6-8aef-591403eccdd5-tls-certs\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.262103 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.262075 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrwh\" (UniqueName: \"kubernetes.io/projected/db19012e-1921-4df6-8aef-591403eccdd5-kube-api-access-nfrwh\") pod \"auth-test-model-826a0f-kserve-5b979988d7-x54bc\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.352443 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.352351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:10:52.505680 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.505657 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc"] Jul 15 20:10:52.507620 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:10:52.507592 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb19012e_1921_4df6_8aef_591403eccdd5.slice/crio-b77d6106e25362c7a140925b4af7f01d7030489a823c86727815cfc7ef507a1e WatchSource:0}: Error finding container b77d6106e25362c7a140925b4af7f01d7030489a823c86727815cfc7ef507a1e: Status 404 returned error can't find the container with id b77d6106e25362c7a140925b4af7f01d7030489a823c86727815cfc7ef507a1e Jul 15 20:10:52.880964 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.880924 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" event={"ID":"db19012e-1921-4df6-8aef-591403eccdd5","Type":"ContainerStarted","Data":"eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b"} Jul 15 20:10:52.881138 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.880978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" event={"ID":"db19012e-1921-4df6-8aef-591403eccdd5","Type":"ContainerStarted","Data":"b77d6106e25362c7a140925b4af7f01d7030489a823c86727815cfc7ef507a1e"} Jul 15 20:10:52.897048 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:52.896998 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" podStartSLOduration=0.896983696 podStartE2EDuration="896.983696ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:10:52.895396857 +0000 UTC m=+2534.756601209" watchObservedRunningTime="2026-07-15 20:10:52.896983696 +0000 UTC m=+2534.758188049" Jul 15 20:10:57.637691 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:10:57.637650 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:11:02.352714 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:02.352680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:11:02.353201 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:02.352911 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:11:02.365418 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:02.365395 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:11:02.953639 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:02.953601 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:11:17.920871 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:17.920826 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l"] Jul 15 20:11:17.924254 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:17.924234 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:17.927084 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:17.927062 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"default-dockercfg-5bjjt\"" Jul 15 20:11:17.927084 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:17.927075 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"auth-test-model-b72424-kserve-self-signed-certs\"" Jul 15 20:11:17.927268 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:17.927062 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"openshift-service-ca.crt\"" Jul 15 20:11:17.927268 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:17.927085 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"kube-root-ca.crt\"" Jul 15 20:11:17.932618 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:17.932596 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l"] Jul 15 20:11:18.001728 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.001687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jz4\" (UniqueName: \"kubernetes.io/projected/292701c5-549c-4cbc-bfd8-f09d86e44173-kube-api-access-x4jz4\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.001728 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.001733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-home\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.002027 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.001752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-dshm\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.002027 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.001843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-tmp-dir\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.002027 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.001884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-model-cache\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.002027 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.001984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292701c5-549c-4cbc-bfd8-f09d86e44173-tls-certs\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.102703 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.102665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-tmp-dir\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.102703 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.102705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-model-cache\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.103062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.102791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292701c5-549c-4cbc-bfd8-f09d86e44173-tls-certs\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.103062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.102899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jz4\" (UniqueName: \"kubernetes.io/projected/292701c5-549c-4cbc-bfd8-f09d86e44173-kube-api-access-x4jz4\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.103062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.102926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-home\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.103062 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.102945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-dshm\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.103634 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.103522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-model-cache\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.103793 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.103635 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-tmp-dir\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.103793 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.103678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-home\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.105868 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.105847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-dshm\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.105973 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.105876 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292701c5-549c-4cbc-bfd8-f09d86e44173-tls-certs\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.112283 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.112261 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jz4\" (UniqueName: \"kubernetes.io/projected/292701c5-549c-4cbc-bfd8-f09d86e44173-kube-api-access-x4jz4\") pod \"auth-test-model-b72424-kserve-b5bfd6f5-5xh4l\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.237336 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.236961 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:18.382197 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:18.382165 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l"] Jul 15 20:11:18.384000 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:11:18.383971 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod292701c5_549c_4cbc_bfd8_f09d86e44173.slice/crio-58c9a8bfca44a61eaef7d06c444d1232e14cbd0c8469557cca706e2325e7b818 WatchSource:0}: Error finding container 58c9a8bfca44a61eaef7d06c444d1232e14cbd0c8469557cca706e2325e7b818: Status 404 returned error can't find the container with id 58c9a8bfca44a61eaef7d06c444d1232e14cbd0c8469557cca706e2325e7b818 Jul 15 20:11:19.011484 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:19.011449 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" event={"ID":"292701c5-549c-4cbc-bfd8-f09d86e44173","Type":"ContainerStarted","Data":"562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e"} Jul 15 20:11:19.011915 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:19.011493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" event={"ID":"292701c5-549c-4cbc-bfd8-f09d86e44173","Type":"ContainerStarted","Data":"58c9a8bfca44a61eaef7d06c444d1232e14cbd0c8469557cca706e2325e7b818"} Jul 15 20:11:19.028800 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:19.028746 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" podStartSLOduration=2.028727859 podStartE2EDuration="2.028727859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:11:19.027275107 +0000 UTC m=+2560.888479460" watchObservedRunningTime="2026-07-15 20:11:19.028727859 +0000 UTC m=+2560.889932212" Jul 15 20:11:21.828283 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:21.828242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:11:28.237592 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:28.237550 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:28.237592 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:28.237601 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:28.249963 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:28.249932 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:29.073797 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:29.073764 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:11:37.834101 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:37.834063 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:11:56.663278 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:11:56.663242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:12:12.836156 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:12:12.836077 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:12:32.734976 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:12:32.734945 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:12:50.434121 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:12:50.434087 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:13:08.228106 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:08.228068 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:13:22.617644 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:22.617609 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc"] Jul 15 20:13:24.042298 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:24.042265 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l"] Jul 15 20:13:25.740731 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:25.740697 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:13:37.638228 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:37.638182 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" podUID="db19012e-1921-4df6-8aef-591403eccdd5" containerName="main" containerID="cri-o://eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b" gracePeriod=45 Jul 15 20:13:37.904529 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:37.904502 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:13:38.034303 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034263 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfrwh\" (UniqueName: \"kubernetes.io/projected/db19012e-1921-4df6-8aef-591403eccdd5-kube-api-access-nfrwh\") pod \"db19012e-1921-4df6-8aef-591403eccdd5\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " Jul 15 20:13:38.034303 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034306 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-dshm\") pod \"db19012e-1921-4df6-8aef-591403eccdd5\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " Jul 15 20:13:38.034562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034342 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db19012e-1921-4df6-8aef-591403eccdd5-tls-certs\") pod \"db19012e-1921-4df6-8aef-591403eccdd5\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " Jul 15 20:13:38.034562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034385 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-tmp-dir\") pod \"db19012e-1921-4df6-8aef-591403eccdd5\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " Jul 15 20:13:38.034562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034473 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-model-cache\") pod \"db19012e-1921-4df6-8aef-591403eccdd5\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " Jul 15 20:13:38.034562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034516 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-home\") pod \"db19012e-1921-4df6-8aef-591403eccdd5\" (UID: \"db19012e-1921-4df6-8aef-591403eccdd5\") " Jul 15 20:13:38.034791 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034694 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "db19012e-1921-4df6-8aef-591403eccdd5" (UID: "db19012e-1921-4df6-8aef-591403eccdd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:38.034791 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034730 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db19012e-1921-4df6-8aef-591403eccdd5" (UID: "db19012e-1921-4df6-8aef-591403eccdd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:38.034791 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034766 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-home" (OuterVolumeSpecName: "home") pod "db19012e-1921-4df6-8aef-591403eccdd5" (UID: "db19012e-1921-4df6-8aef-591403eccdd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:38.034976 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034954 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-tmp-dir\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:38.035015 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034978 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-model-cache\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:38.035015 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.034994 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-home\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:38.036856 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.036829 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-dshm" (OuterVolumeSpecName: "dshm") pod "db19012e-1921-4df6-8aef-591403eccdd5" (UID: "db19012e-1921-4df6-8aef-591403eccdd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:38.036973 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.036899 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db19012e-1921-4df6-8aef-591403eccdd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db19012e-1921-4df6-8aef-591403eccdd5" (UID: "db19012e-1921-4df6-8aef-591403eccdd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:13:38.036973 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.036954 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db19012e-1921-4df6-8aef-591403eccdd5-kube-api-access-nfrwh" (OuterVolumeSpecName: "kube-api-access-nfrwh") pod "db19012e-1921-4df6-8aef-591403eccdd5" (UID: "db19012e-1921-4df6-8aef-591403eccdd5"). InnerVolumeSpecName "kube-api-access-nfrwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:13:38.136182 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.136131 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrwh\" (UniqueName: \"kubernetes.io/projected/db19012e-1921-4df6-8aef-591403eccdd5-kube-api-access-nfrwh\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:38.136182 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.136174 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db19012e-1921-4df6-8aef-591403eccdd5-dshm\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:38.136182 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.136191 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db19012e-1921-4df6-8aef-591403eccdd5-tls-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:38.722985 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.722950 2573 generic.go:356] "Generic (PLEG): container finished" podID="db19012e-1921-4df6-8aef-591403eccdd5" containerID="eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b" exitCode=0 Jul 15 20:13:38.723377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.723005 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" Jul 15 20:13:38.723377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.723033 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" event={"ID":"db19012e-1921-4df6-8aef-591403eccdd5","Type":"ContainerDied","Data":"eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b"} Jul 15 20:13:38.723377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.723075 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc" event={"ID":"db19012e-1921-4df6-8aef-591403eccdd5","Type":"ContainerDied","Data":"b77d6106e25362c7a140925b4af7f01d7030489a823c86727815cfc7ef507a1e"} Jul 15 20:13:38.723377 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.723092 2573 scope.go:117] "RemoveContainer" containerID="eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b" Jul 15 20:13:38.732271 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.732247 2573 scope.go:117] "RemoveContainer" containerID="eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b" Jul 15 20:13:38.732512 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:13:38.732495 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b\": container with ID starting with eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b not found: ID does not exist" containerID="eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b" Jul 15 20:13:38.732566 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.732520 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b"} err="failed to get container status \"eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b\": rpc error: code = NotFound desc = could not find container \"eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b\": container with ID starting with eb767cef1596e3606a66f5b78e8ad8b03f8962c07c7df4a65afc932db960649b not found: ID does not exist" Jul 15 20:13:38.738576 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.738553 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc"] Jul 15 20:13:38.742166 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.742145 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/auth-test-model-826a0f-kserve-5b979988d7-x54bc"] Jul 15 20:13:38.815883 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.815860 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 20:13:38.823275 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.823256 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 20:13:38.823736 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.823717 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 20:13:38.830312 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:38.830293 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6qp4s_61bd4882-80fa-43e2-9dab-ad2dd5ef25d8/ovn-acl-logging/0.log" Jul 15 20:13:39.063653 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.063549 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" podUID="292701c5-549c-4cbc-bfd8-f09d86e44173" containerName="main" containerID="cri-o://562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e" gracePeriod=45 Jul 15 20:13:39.067702 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.067675 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" podUID="292701c5-549c-4cbc-bfd8-f09d86e44173" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": EOF" Jul 15 20:13:39.331510 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.331486 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:13:39.449286 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449256 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292701c5-549c-4cbc-bfd8-f09d86e44173-tls-certs\") pod \"292701c5-549c-4cbc-bfd8-f09d86e44173\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " Jul 15 20:13:39.449460 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449329 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-dshm\") pod \"292701c5-549c-4cbc-bfd8-f09d86e44173\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " Jul 15 20:13:39.449460 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449365 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-tmp-dir\") pod \"292701c5-549c-4cbc-bfd8-f09d86e44173\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " Jul 15 20:13:39.449460 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449396 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-home\") pod \"292701c5-549c-4cbc-bfd8-f09d86e44173\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " Jul 15 20:13:39.449460 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449432 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-model-cache\") pod \"292701c5-549c-4cbc-bfd8-f09d86e44173\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " Jul 15 20:13:39.449685 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449491 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4jz4\" (UniqueName: \"kubernetes.io/projected/292701c5-549c-4cbc-bfd8-f09d86e44173-kube-api-access-x4jz4\") pod \"292701c5-549c-4cbc-bfd8-f09d86e44173\" (UID: \"292701c5-549c-4cbc-bfd8-f09d86e44173\") " Jul 15 20:13:39.449685 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449651 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "292701c5-549c-4cbc-bfd8-f09d86e44173" (UID: "292701c5-549c-4cbc-bfd8-f09d86e44173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:39.449789 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449697 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-home" (OuterVolumeSpecName: "home") pod "292701c5-549c-4cbc-bfd8-f09d86e44173" (UID: "292701c5-549c-4cbc-bfd8-f09d86e44173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:39.449789 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449755 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-model-cache" (OuterVolumeSpecName: "model-cache") pod "292701c5-549c-4cbc-bfd8-f09d86e44173" (UID: "292701c5-549c-4cbc-bfd8-f09d86e44173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:39.449932 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449861 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-tmp-dir\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:39.449932 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.449886 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-home\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:39.451793 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.451771 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-dshm" (OuterVolumeSpecName: "dshm") pod "292701c5-549c-4cbc-bfd8-f09d86e44173" (UID: "292701c5-549c-4cbc-bfd8-f09d86e44173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:13:39.451912 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.451869 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292701c5-549c-4cbc-bfd8-f09d86e44173-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "292701c5-549c-4cbc-bfd8-f09d86e44173" (UID: "292701c5-549c-4cbc-bfd8-f09d86e44173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:13:39.451981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.451936 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292701c5-549c-4cbc-bfd8-f09d86e44173-kube-api-access-x4jz4" (OuterVolumeSpecName: "kube-api-access-x4jz4") pod "292701c5-549c-4cbc-bfd8-f09d86e44173" (UID: "292701c5-549c-4cbc-bfd8-f09d86e44173"). InnerVolumeSpecName "kube-api-access-x4jz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:13:39.550470 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.550442 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-model-cache\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:39.550470 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.550465 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jz4\" (UniqueName: \"kubernetes.io/projected/292701c5-549c-4cbc-bfd8-f09d86e44173-kube-api-access-x4jz4\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:39.550470 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.550474 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292701c5-549c-4cbc-bfd8-f09d86e44173-tls-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:39.550701 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.550483 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292701c5-549c-4cbc-bfd8-f09d86e44173-dshm\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:13:39.730009 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.729974 2573 generic.go:356] "Generic (PLEG): container finished" podID="292701c5-549c-4cbc-bfd8-f09d86e44173" containerID="562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e" exitCode=0 Jul 15 20:13:39.730429 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.730020 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" Jul 15 20:13:39.730429 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.730037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" event={"ID":"292701c5-549c-4cbc-bfd8-f09d86e44173","Type":"ContainerDied","Data":"562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e"} Jul 15 20:13:39.730429 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.730082 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l" event={"ID":"292701c5-549c-4cbc-bfd8-f09d86e44173","Type":"ContainerDied","Data":"58c9a8bfca44a61eaef7d06c444d1232e14cbd0c8469557cca706e2325e7b818"} Jul 15 20:13:39.730429 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.730102 2573 scope.go:117] "RemoveContainer" containerID="562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e" Jul 15 20:13:39.739844 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.739832 2573 scope.go:117] "RemoveContainer" containerID="562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e" Jul 15 20:13:39.740130 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:13:39.740107 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e\": container with ID starting with 562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e not found: ID does not exist" containerID="562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e" Jul 15 20:13:39.740198 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.740144 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e"} err="failed to get container status \"562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e\": rpc error: code = NotFound desc = could not find container \"562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e\": container with ID starting with 562cd25479eb57f34328c91517726d621edd9121ef703d4d96e64b956a48ce1e not found: ID does not exist" Jul 15 20:13:39.749000 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.748981 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l"] Jul 15 20:13:39.751629 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:39.751608 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/auth-test-model-b72424-kserve-b5bfd6f5-5xh4l"] Jul 15 20:13:40.629106 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:40.629073 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:13:40.697263 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:40.697233 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292701c5-549c-4cbc-bfd8-f09d86e44173" path="/var/lib/kubelet/pods/292701c5-549c-4cbc-bfd8-f09d86e44173/volumes" Jul 15 20:13:40.697703 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:40.697687 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db19012e-1921-4df6-8aef-591403eccdd5" path="/var/lib/kubelet/pods/db19012e-1921-4df6-8aef-591403eccdd5/volumes" Jul 15 20:13:44.930971 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.930930 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5"] Jul 15 20:13:44.931382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.931309 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db19012e-1921-4df6-8aef-591403eccdd5" containerName="main" Jul 15 20:13:44.931382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.931323 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="db19012e-1921-4df6-8aef-591403eccdd5" containerName="main" Jul 15 20:13:44.931382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.931335 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292701c5-549c-4cbc-bfd8-f09d86e44173" containerName="main" Jul 15 20:13:44.931382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.931341 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="292701c5-549c-4cbc-bfd8-f09d86e44173" containerName="main" Jul 15 20:13:44.931555 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.931407 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="292701c5-549c-4cbc-bfd8-f09d86e44173" containerName="main" Jul 15 20:13:44.931555 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.931427 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="db19012e-1921-4df6-8aef-591403eccdd5" containerName="main" Jul 15 20:13:44.935954 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.935934 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:44.938321 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.938302 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"openshift-service-ca.crt\"" Jul 15 20:13:44.938995 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.938974 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"sub-test-model-b72424-kserve-self-signed-certs\"" Jul 15 20:13:44.939260 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.939241 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"default-dockercfg-5bjjt\"" Jul 15 20:13:44.939367 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.939275 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"kube-root-ca.crt\"" Jul 15 20:13:44.944891 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:44.944865 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5"] Jul 15 20:13:45.007380 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.007352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-home\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.007583 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.007400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-model-cache\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.007583 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.007429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-dshm\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.007583 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.007531 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897s4\" (UniqueName: \"kubernetes.io/projected/2c40bafc-6746-4601-b859-298db4311293-kube-api-access-897s4\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.007583 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.007569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c40bafc-6746-4601-b859-298db4311293-tls-certs\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.007795 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.007624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-tmp-dir\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.108532 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.108493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-dshm\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.108532 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.108547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897s4\" (UniqueName: \"kubernetes.io/projected/2c40bafc-6746-4601-b859-298db4311293-kube-api-access-897s4\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112086 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.108662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c40bafc-6746-4601-b859-298db4311293-tls-certs\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112086 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.109179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-tmp-dir\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112086 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.109304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-home\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112086 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.109377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-model-cache\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112086 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.109769 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-model-cache\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112086 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.111935 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-home\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112086 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.111975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-tmp-dir\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112616 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.112112 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-dshm\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.112681 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.112623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c40bafc-6746-4601-b859-298db4311293-tls-certs\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.123835 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.123777 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-897s4\" (UniqueName: \"kubernetes.io/projected/2c40bafc-6746-4601-b859-298db4311293-kube-api-access-897s4\") pod \"sub-test-model-b72424-kserve-777bd9fb49-45hx5\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.245604 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.245516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:45.393776 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.393748 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5"] Jul 15 20:13:45.395659 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:13:45.395635 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c40bafc_6746_4601_b859_298db4311293.slice/crio-bdead05001831401128d94753257b3c39e6a2398afc4712447beccb16000b128 WatchSource:0}: Error finding container bdead05001831401128d94753257b3c39e6a2398afc4712447beccb16000b128: Status 404 returned error can't find the container with id bdead05001831401128d94753257b3c39e6a2398afc4712447beccb16000b128 Jul 15 20:13:45.762569 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.762533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" event={"ID":"2c40bafc-6746-4601-b859-298db4311293","Type":"ContainerStarted","Data":"61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d"} Jul 15 20:13:45.762748 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.762584 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" event={"ID":"2c40bafc-6746-4601-b859-298db4311293","Type":"ContainerStarted","Data":"bdead05001831401128d94753257b3c39e6a2398afc4712447beccb16000b128"} Jul 15 20:13:45.779340 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:45.779278 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" podStartSLOduration=1.779254112 podStartE2EDuration="1.779254112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:13:45.777371032 +0000 UTC m=+2707.638575385" watchObservedRunningTime="2026-07-15 20:13:45.779254112 +0000 UTC m=+2707.640458466" Jul 15 20:13:49.235995 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:49.235964 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:13:55.245651 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:55.245612 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:55.246174 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:55.245667 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:55.259061 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:55.259036 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:13:55.823912 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:13:55.823882 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:14:00.728045 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:00.728010 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:14:17.336301 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:17.336261 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5"] Jul 15 20:14:20.027390 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:20.027347 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:14:31.412412 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.412370 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m"] Jul 15 20:14:31.416112 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.416090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.419097 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.419068 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"rate-test-model-826a0f-kserve-self-signed-certs\"" Jul 15 20:14:31.419232 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.419107 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"default-dockercfg-ccrm8\"" Jul 15 20:14:31.419232 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.419116 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"openshift-service-ca.crt\"" Jul 15 20:14:31.419232 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.419117 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-826a0f\"/\"kube-root-ca.crt\"" Jul 15 20:14:31.422945 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.422919 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m"] Jul 15 20:14:31.541139 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.541103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-home\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.541312 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.541147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-dshm\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.541312 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.541230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-model-cache\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.541312 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.541288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-tmp-dir\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.541419 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.541335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5110e51f-2488-4414-a065-494d04ee22c4-tls-certs\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.541419 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.541357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtc6\" (UniqueName: \"kubernetes.io/projected/5110e51f-2488-4414-a065-494d04ee22c4-kube-api-access-ljtc6\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642170 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-dshm\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642358 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-model-cache\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642358 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-tmp-dir\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642358 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5110e51f-2488-4414-a065-494d04ee22c4-tls-certs\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642358 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtc6\" (UniqueName: \"kubernetes.io/projected/5110e51f-2488-4414-a065-494d04ee22c4-kube-api-access-ljtc6\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642595 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-home\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642652 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-model-cache\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642710 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-home\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.642933 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.642909 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-tmp-dir\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.645217 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.645190 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-dshm\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.645477 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.645456 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5110e51f-2488-4414-a065-494d04ee22c4-tls-certs\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.650175 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.650155 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtc6\" (UniqueName: \"kubernetes.io/projected/5110e51f-2488-4414-a065-494d04ee22c4-kube-api-access-ljtc6\") pod \"rate-test-model-826a0f-kserve-7f4b487f99-7kk4m\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.727484 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.727390 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:31.863968 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:31.863944 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m"] Jul 15 20:14:31.865380 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:14:31.865332 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5110e51f_2488_4414_a065_494d04ee22c4.slice/crio-e86f13394c8116a59977f452c507794f92cc596bf201fe0773c9dd230a6f8edf WatchSource:0}: Error finding container e86f13394c8116a59977f452c507794f92cc596bf201fe0773c9dd230a6f8edf: Status 404 returned error can't find the container with id e86f13394c8116a59977f452c507794f92cc596bf201fe0773c9dd230a6f8edf Jul 15 20:14:32.007872 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.007754 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" event={"ID":"5110e51f-2488-4414-a065-494d04ee22c4","Type":"ContainerStarted","Data":"ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7"} Jul 15 20:14:32.007872 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.007822 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" event={"ID":"5110e51f-2488-4414-a065-494d04ee22c4","Type":"ContainerStarted","Data":"e86f13394c8116a59977f452c507794f92cc596bf201fe0773c9dd230a6f8edf"} Jul 15 20:14:32.024519 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.024471 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" podStartSLOduration=1.024456561 podStartE2EDuration="1.024456561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:14:32.022274944 +0000 UTC m=+2753.883479297" watchObservedRunningTime="2026-07-15 20:14:32.024456561 +0000 UTC m=+2753.885660939" Jul 15 20:14:32.358145 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.358053 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" podUID="2c40bafc-6746-4601-b859-298db4311293" containerName="main" containerID="cri-o://61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d" gracePeriod=45 Jul 15 20:14:32.601696 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.601672 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:14:32.752980 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.752944 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-dshm\") pod \"2c40bafc-6746-4601-b859-298db4311293\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " Jul 15 20:14:32.753176 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753015 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-897s4\" (UniqueName: \"kubernetes.io/projected/2c40bafc-6746-4601-b859-298db4311293-kube-api-access-897s4\") pod \"2c40bafc-6746-4601-b859-298db4311293\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " Jul 15 20:14:32.753176 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753061 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-tmp-dir\") pod \"2c40bafc-6746-4601-b859-298db4311293\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " Jul 15 20:14:32.753176 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753160 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c40bafc-6746-4601-b859-298db4311293-tls-certs\") pod \"2c40bafc-6746-4601-b859-298db4311293\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " Jul 15 20:14:32.753345 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753230 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-model-cache\") pod \"2c40bafc-6746-4601-b859-298db4311293\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " Jul 15 20:14:32.753345 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753259 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-home\") pod \"2c40bafc-6746-4601-b859-298db4311293\" (UID: \"2c40bafc-6746-4601-b859-298db4311293\") " Jul 15 20:14:32.753445 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753375 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c40bafc-6746-4601-b859-298db4311293" (UID: "2c40bafc-6746-4601-b859-298db4311293"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:14:32.753506 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753454 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c40bafc-6746-4601-b859-298db4311293" (UID: "2c40bafc-6746-4601-b859-298db4311293"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:14:32.753599 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753577 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-home" (OuterVolumeSpecName: "home") pod "2c40bafc-6746-4601-b859-298db4311293" (UID: "2c40bafc-6746-4601-b859-298db4311293"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:14:32.753654 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753605 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-model-cache\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:14:32.753654 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.753623 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-tmp-dir\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:14:32.755402 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.755380 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c40bafc-6746-4601-b859-298db4311293-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c40bafc-6746-4601-b859-298db4311293" (UID: "2c40bafc-6746-4601-b859-298db4311293"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:14:32.755672 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.755658 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-dshm" (OuterVolumeSpecName: "dshm") pod "2c40bafc-6746-4601-b859-298db4311293" (UID: "2c40bafc-6746-4601-b859-298db4311293"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:14:32.755734 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.755692 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c40bafc-6746-4601-b859-298db4311293-kube-api-access-897s4" (OuterVolumeSpecName: "kube-api-access-897s4") pod "2c40bafc-6746-4601-b859-298db4311293" (UID: "2c40bafc-6746-4601-b859-298db4311293"). InnerVolumeSpecName "kube-api-access-897s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:14:32.855028 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.854995 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-897s4\" (UniqueName: \"kubernetes.io/projected/2c40bafc-6746-4601-b859-298db4311293-kube-api-access-897s4\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:14:32.855028 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.855019 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c40bafc-6746-4601-b859-298db4311293-tls-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:14:32.855028 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.855029 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-home\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:14:32.855028 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:32.855036 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c40bafc-6746-4601-b859-298db4311293-dshm\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:14:33.015197 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.015157 2573 generic.go:356] "Generic (PLEG): container finished" podID="2c40bafc-6746-4601-b859-298db4311293" containerID="61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d" exitCode=0 Jul 15 20:14:33.015391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.015209 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" Jul 15 20:14:33.015391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.015254 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" event={"ID":"2c40bafc-6746-4601-b859-298db4311293","Type":"ContainerDied","Data":"61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d"} Jul 15 20:14:33.015391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.015295 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5" event={"ID":"2c40bafc-6746-4601-b859-298db4311293","Type":"ContainerDied","Data":"bdead05001831401128d94753257b3c39e6a2398afc4712447beccb16000b128"} Jul 15 20:14:33.015391 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.015329 2573 scope.go:117] "RemoveContainer" containerID="61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d" Jul 15 20:14:33.024688 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.024672 2573 scope.go:117] "RemoveContainer" containerID="61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d" Jul 15 20:14:33.024961 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:14:33.024943 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d\": container with ID starting with 61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d not found: ID does not exist" containerID="61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d" Jul 15 20:14:33.025025 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.024971 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d"} err="failed to get container status \"61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d\": rpc error: code = NotFound desc = could not find container \"61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d\": container with ID starting with 61ea42c50054ed9b06c23c154796e3ae8ef36edd1e05fc52b2788ea972f6b18d not found: ID does not exist" Jul 15 20:14:33.035585 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.035561 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5"] Jul 15 20:14:33.040084 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:33.040063 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/sub-test-model-b72424-kserve-777bd9fb49-45hx5"] Jul 15 20:14:34.696308 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:34.696275 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c40bafc-6746-4601-b859-298db4311293" path="/var/lib/kubelet/pods/2c40bafc-6746-4601-b859-298db4311293/volumes" Jul 15 20:14:35.637689 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:35.637652 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:14:41.728457 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:41.728424 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:41.728955 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:41.728476 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:41.740959 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:41.740934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:42.073696 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:42.073606 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:14:52.332825 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.332774 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms"] Jul 15 20:14:52.333399 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.333378 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c40bafc-6746-4601-b859-298db4311293" containerName="main" Jul 15 20:14:52.333485 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.333403 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c40bafc-6746-4601-b859-298db4311293" containerName="main" Jul 15 20:14:52.333539 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.333495 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c40bafc-6746-4601-b859-298db4311293" containerName="main" Jul 15 20:14:52.336432 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.336411 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.339243 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.339217 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"openshift-service-ca.crt\"" Jul 15 20:14:52.339970 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.339923 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"kube-root-ca.crt\"" Jul 15 20:14:52.340103 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.339935 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"default-dockercfg-5bjjt\"" Jul 15 20:14:52.340103 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.340031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b72424\"/\"rate-test-model-b72424-kserve-self-signed-certs\"" Jul 15 20:14:52.350656 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.350629 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms"] Jul 15 20:14:52.443455 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.443425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkpb\" (UniqueName: \"kubernetes.io/projected/4e33499e-e231-4413-885c-2a95eb162eed-kube-api-access-qwkpb\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.443633 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.443470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-home\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.443633 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.443487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-dshm\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.443633 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.443510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-tmp-dir\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.443756 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.443623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33499e-e231-4413-885c-2a95eb162eed-tls-certs\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.443756 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.443675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-model-cache\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.544841 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.544787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-model-cache\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.544841 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.544849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkpb\" (UniqueName: \"kubernetes.io/projected/4e33499e-e231-4413-885c-2a95eb162eed-kube-api-access-qwkpb\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.545097 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.545020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-home\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.545097 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.545057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-dshm\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.545097 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.545093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-tmp-dir\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.545271 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.545182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33499e-e231-4413-885c-2a95eb162eed-tls-certs\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.545271 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.545256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-model-cache\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.545422 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.545398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-home\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.545495 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.545477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-tmp-dir\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.547518 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.547495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-dshm\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.547646 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.547631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33499e-e231-4413-885c-2a95eb162eed-tls-certs\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.552915 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.552897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkpb\" (UniqueName: \"kubernetes.io/projected/4e33499e-e231-4413-885c-2a95eb162eed-kube-api-access-qwkpb\") pod \"rate-test-model-b72424-kserve-75cff545d7-zhcms\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.646382 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.646295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:14:52.783763 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:52.783737 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms"] Jul 15 20:14:52.786032 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:14:52.786006 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e33499e_e231_4413_885c_2a95eb162eed.slice/crio-906372881b5730c8d93619d3dcd6b58547ecb4ac647c27dd818b07009c9c103b WatchSource:0}: Error finding container 906372881b5730c8d93619d3dcd6b58547ecb4ac647c27dd818b07009c9c103b: Status 404 returned error can't find the container with id 906372881b5730c8d93619d3dcd6b58547ecb4ac647c27dd818b07009c9c103b Jul 15 20:14:53.120360 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:53.120320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" event={"ID":"4e33499e-e231-4413-885c-2a95eb162eed","Type":"ContainerStarted","Data":"bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998"} Jul 15 20:14:53.120360 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:53.120363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" event={"ID":"4e33499e-e231-4413-885c-2a95eb162eed","Type":"ContainerStarted","Data":"906372881b5730c8d93619d3dcd6b58547ecb4ac647c27dd818b07009c9c103b"} Jul 15 20:14:53.138862 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:53.138819 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" podStartSLOduration=1.138789537 podStartE2EDuration="1.138789537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:14:53.137132966 +0000 UTC m=+2774.998337325" watchObservedRunningTime="2026-07-15 20:14:53.138789537 +0000 UTC m=+2774.999993913" Jul 15 20:14:56.234819 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:14:56.234769 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:15:02.647189 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:02.647101 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:15:02.647651 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:02.647439 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:15:02.659681 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:02.659660 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:15:03.182340 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:03.182297 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:15:14.541502 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:14.541465 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:15:29.831739 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:29.831705 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:15:36.919452 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:36.919414 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m"] Jul 15 20:15:37.821297 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:37.821268 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms"] Jul 15 20:15:43.837122 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:43.837087 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:15:51.942826 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:51.942751 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" podUID="5110e51f-2488-4414-a065-494d04ee22c4" containerName="main" containerID="cri-o://ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7" gracePeriod=45 Jul 15 20:15:52.063460 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.063423 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" podUID="5110e51f-2488-4414-a065-494d04ee22c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused" Jul 15 20:15:52.201116 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.201046 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:15:52.305291 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305187 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-home\") pod \"5110e51f-2488-4414-a065-494d04ee22c4\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " Jul 15 20:15:52.305466 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305319 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-tmp-dir\") pod \"5110e51f-2488-4414-a065-494d04ee22c4\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " Jul 15 20:15:52.305466 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305363 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5110e51f-2488-4414-a065-494d04ee22c4-tls-certs\") pod \"5110e51f-2488-4414-a065-494d04ee22c4\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " Jul 15 20:15:52.305466 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305389 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-home" (OuterVolumeSpecName: "home") pod "5110e51f-2488-4414-a065-494d04ee22c4" (UID: "5110e51f-2488-4414-a065-494d04ee22c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:52.305466 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305454 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-model-cache\") pod \"5110e51f-2488-4414-a065-494d04ee22c4\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " Jul 15 20:15:52.305609 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305492 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-dshm\") pod \"5110e51f-2488-4414-a065-494d04ee22c4\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " Jul 15 20:15:52.305609 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305552 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljtc6\" (UniqueName: \"kubernetes.io/projected/5110e51f-2488-4414-a065-494d04ee22c4-kube-api-access-ljtc6\") pod \"5110e51f-2488-4414-a065-494d04ee22c4\" (UID: \"5110e51f-2488-4414-a065-494d04ee22c4\") " Jul 15 20:15:52.305609 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305558 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5110e51f-2488-4414-a065-494d04ee22c4" (UID: "5110e51f-2488-4414-a065-494d04ee22c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:52.305723 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305699 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "5110e51f-2488-4414-a065-494d04ee22c4" (UID: "5110e51f-2488-4414-a065-494d04ee22c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:52.305913 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305891 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-model-cache\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:52.305913 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305910 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-home\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:52.305913 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.305918 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-tmp-dir\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:52.307775 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.307752 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-dshm" (OuterVolumeSpecName: "dshm") pod "5110e51f-2488-4414-a065-494d04ee22c4" (UID: "5110e51f-2488-4414-a065-494d04ee22c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:52.307863 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.307773 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5110e51f-2488-4414-a065-494d04ee22c4-kube-api-access-ljtc6" (OuterVolumeSpecName: "kube-api-access-ljtc6") pod "5110e51f-2488-4414-a065-494d04ee22c4" (UID: "5110e51f-2488-4414-a065-494d04ee22c4"). InnerVolumeSpecName "kube-api-access-ljtc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:15:52.307904 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.307859 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5110e51f-2488-4414-a065-494d04ee22c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5110e51f-2488-4414-a065-494d04ee22c4" (UID: "5110e51f-2488-4414-a065-494d04ee22c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:15:52.406514 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.406478 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5110e51f-2488-4414-a065-494d04ee22c4-tls-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:52.406514 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.406511 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5110e51f-2488-4414-a065-494d04ee22c4-dshm\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:52.406514 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.406520 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtc6\" (UniqueName: \"kubernetes.io/projected/5110e51f-2488-4414-a065-494d04ee22c4-kube-api-access-ljtc6\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:52.418343 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.418314 2573 generic.go:356] "Generic (PLEG): container finished" podID="5110e51f-2488-4414-a065-494d04ee22c4" containerID="ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7" exitCode=0 Jul 15 20:15:52.418497 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.418350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" event={"ID":"5110e51f-2488-4414-a065-494d04ee22c4","Type":"ContainerDied","Data":"ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7"} Jul 15 20:15:52.418497 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.418358 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" Jul 15 20:15:52.418497 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.418377 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m" event={"ID":"5110e51f-2488-4414-a065-494d04ee22c4","Type":"ContainerDied","Data":"e86f13394c8116a59977f452c507794f92cc596bf201fe0773c9dd230a6f8edf"} Jul 15 20:15:52.418497 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.418395 2573 scope.go:117] "RemoveContainer" containerID="ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7" Jul 15 20:15:52.428484 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.428464 2573 scope.go:117] "RemoveContainer" containerID="ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7" Jul 15 20:15:52.428770 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:15:52.428750 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7\": container with ID starting with ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7 not found: ID does not exist" containerID="ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7" Jul 15 20:15:52.428903 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.428776 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7"} err="failed to get container status \"ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7\": rpc error: code = NotFound desc = could not find container \"ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7\": container with ID starting with ebd6fac721d4274b126e9ca9e3df2999f3a76f1468d59b8d6b123d877827f5d7 not found: ID does not exist" Jul 15 20:15:52.439489 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.439469 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m"] Jul 15 20:15:52.443245 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.443223 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-826a0f/rate-test-model-826a0f-kserve-7f4b487f99-7kk4m"] Jul 15 20:15:52.696162 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.696129 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5110e51f-2488-4414-a065-494d04ee22c4" path="/var/lib/kubelet/pods/5110e51f-2488-4414-a065-494d04ee22c4/volumes" Jul 15 20:15:52.842356 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:52.842309 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" podUID="4e33499e-e231-4413-885c-2a95eb162eed" containerName="main" containerID="cri-o://bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998" gracePeriod=45 Jul 15 20:15:53.096689 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.096664 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:15:53.213735 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.213641 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-model-cache\") pod \"4e33499e-e231-4413-885c-2a95eb162eed\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " Jul 15 20:15:53.213735 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.213685 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33499e-e231-4413-885c-2a95eb162eed-tls-certs\") pod \"4e33499e-e231-4413-885c-2a95eb162eed\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " Jul 15 20:15:53.213735 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.213713 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwkpb\" (UniqueName: \"kubernetes.io/projected/4e33499e-e231-4413-885c-2a95eb162eed-kube-api-access-qwkpb\") pod \"4e33499e-e231-4413-885c-2a95eb162eed\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " Jul 15 20:15:53.214050 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.213752 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-home\") pod \"4e33499e-e231-4413-885c-2a95eb162eed\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " Jul 15 20:15:53.214050 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.213880 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-dshm\") pod \"4e33499e-e231-4413-885c-2a95eb162eed\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " Jul 15 20:15:53.214050 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.213960 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-tmp-dir\") pod \"4e33499e-e231-4413-885c-2a95eb162eed\" (UID: \"4e33499e-e231-4413-885c-2a95eb162eed\") " Jul 15 20:15:53.214050 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.213963 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e33499e-e231-4413-885c-2a95eb162eed" (UID: "4e33499e-e231-4413-885c-2a95eb162eed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:53.214230 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.214112 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-home" (OuterVolumeSpecName: "home") pod "4e33499e-e231-4413-885c-2a95eb162eed" (UID: "4e33499e-e231-4413-885c-2a95eb162eed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:53.214283 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.214261 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e33499e-e231-4413-885c-2a95eb162eed" (UID: "4e33499e-e231-4413-885c-2a95eb162eed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:53.214351 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.214331 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-model-cache\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:53.214404 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.214356 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-home\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:53.216106 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.216077 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-dshm" (OuterVolumeSpecName: "dshm") pod "4e33499e-e231-4413-885c-2a95eb162eed" (UID: "4e33499e-e231-4413-885c-2a95eb162eed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:15:53.216106 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.216089 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e33499e-e231-4413-885c-2a95eb162eed-kube-api-access-qwkpb" (OuterVolumeSpecName: "kube-api-access-qwkpb") pod "4e33499e-e231-4413-885c-2a95eb162eed" (UID: "4e33499e-e231-4413-885c-2a95eb162eed"). InnerVolumeSpecName "kube-api-access-qwkpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:15:53.216282 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.216158 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e33499e-e231-4413-885c-2a95eb162eed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e33499e-e231-4413-885c-2a95eb162eed" (UID: "4e33499e-e231-4413-885c-2a95eb162eed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:15:53.315867 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.315800 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e33499e-e231-4413-885c-2a95eb162eed-tls-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:53.315867 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.315859 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkpb\" (UniqueName: \"kubernetes.io/projected/4e33499e-e231-4413-885c-2a95eb162eed-kube-api-access-qwkpb\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:53.315867 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.315874 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-dshm\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:53.316107 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.315885 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e33499e-e231-4413-885c-2a95eb162eed-tmp-dir\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:15:53.425995 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.425967 2573 generic.go:356] "Generic (PLEG): container finished" podID="4e33499e-e231-4413-885c-2a95eb162eed" containerID="bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998" exitCode=0 Jul 15 20:15:53.426173 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.426013 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" Jul 15 20:15:53.426173 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.426053 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" event={"ID":"4e33499e-e231-4413-885c-2a95eb162eed","Type":"ContainerDied","Data":"bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998"} Jul 15 20:15:53.426173 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.426101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms" event={"ID":"4e33499e-e231-4413-885c-2a95eb162eed","Type":"ContainerDied","Data":"906372881b5730c8d93619d3dcd6b58547ecb4ac647c27dd818b07009c9c103b"} Jul 15 20:15:53.426173 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.426123 2573 scope.go:117] "RemoveContainer" containerID="bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998" Jul 15 20:15:53.435404 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.435378 2573 scope.go:117] "RemoveContainer" containerID="bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998" Jul 15 20:15:53.435682 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:15:53.435661 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998\": container with ID starting with bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998 not found: ID does not exist" containerID="bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998" Jul 15 20:15:53.435734 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.435693 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998"} err="failed to get container status \"bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998\": rpc error: code = NotFound desc = could not find container \"bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998\": container with ID starting with bb794bbb8ee93821bcfa6b116aedd79d508e9317bd9f8b974876cc0cbe953998 not found: ID does not exist" Jul 15 20:15:53.446666 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.446635 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms"] Jul 15 20:15:53.448389 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:53.448371 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b72424/rate-test-model-b72424-kserve-75cff545d7-zhcms"] Jul 15 20:15:54.695776 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:15:54.695743 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e33499e-e231-4413-885c-2a95eb162eed" path="/var/lib/kubelet/pods/4e33499e-e231-4413-885c-2a95eb162eed/volumes" Jul 15 20:16:10.232890 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:10.232855 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:16:18.410352 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:18.410317 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n"] Jul 15 20:16:18.410761 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:18.410559 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" containerID="cri-o://297bfc7eed3869885c1931b694338d51b1d8a8bfe4ab1616bbbf403e1908081d" gracePeriod=30 Jul 15 20:16:23.575343 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.575294 2573 generic.go:356] "Generic (PLEG): container finished" podID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerID="297bfc7eed3869885c1931b694338d51b1d8a8bfe4ab1616bbbf403e1908081d" exitCode=0 Jul 15 20:16:23.575722 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.575367 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" event={"ID":"40a5cf3d-0db5-4d51-b76c-5fd759a5e647","Type":"ContainerDied","Data":"297bfc7eed3869885c1931b694338d51b1d8a8bfe4ab1616bbbf403e1908081d"} Jul 15 20:16:23.575722 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.575462 2573 scope.go:117] "RemoveContainer" containerID="cf43d20d7d868e657e73b82ce01dac1cea69ad7bee6ed09249376913c13b46b6" Jul 15 20:16:23.666310 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.666287 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:16:23.701487 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701405 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-token\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701487 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701456 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-certs\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701494 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-data\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701524 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-envoy\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701545 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-podinfo\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701593 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szp7m\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-kube-api-access-szp7m\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701619 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-credential-socket\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701651 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istiod-ca-cert\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.701697 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701684 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-socket\") pod \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\" (UID: \"40a5cf3d-0db5-4d51-b76c-5fd759a5e647\") " Jul 15 20:16:23.702073 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.701896 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:23.702183 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.702160 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.702567 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.702539 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-data" (OuterVolumeSpecName: "istio-data") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:23.702664 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.702583 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:16:23.702944 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.702919 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:23.703087 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.703057 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:23.704265 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.704235 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:23.704399 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.704380 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-token" (OuterVolumeSpecName: "istio-token") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:16:23.704650 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.704634 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-kube-api-access-szp7m" (OuterVolumeSpecName: "kube-api-access-szp7m") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "kube-api-access-szp7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:16:23.704737 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.704719 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40a5cf3d-0db5-4d51-b76c-5fd759a5e647" (UID: "40a5cf3d-0db5-4d51-b76c-5fd759a5e647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:16:23.802981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.802940 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.802981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.802973 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.802981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.802982 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.802981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.802990 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szp7m\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-kube-api-access-szp7m\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.803245 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.803000 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.803245 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.803008 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.803245 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.803016 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:23.803245 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:23.803023 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5cf3d-0db5-4d51-b76c-5fd759a5e647-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:24.583499 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:24.583465 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" Jul 15 20:16:24.583974 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:24.583501 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n" event={"ID":"40a5cf3d-0db5-4d51-b76c-5fd759a5e647","Type":"ContainerDied","Data":"304eac972bcaeb897ed89fe858448b546815df40c48484c05d6868b58852deed"} Jul 15 20:16:24.583974 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:24.583571 2573 scope.go:117] "RemoveContainer" containerID="297bfc7eed3869885c1931b694338d51b1d8a8bfe4ab1616bbbf403e1908081d" Jul 15 20:16:24.603962 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:24.603938 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n"] Jul 15 20:16:24.607404 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:24.607370 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-826a0f-openshift-default-b54955b79-m9m9n"] Jul 15 20:16:24.696068 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:24.696032 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" path="/var/lib/kubelet/pods/40a5cf3d-0db5-4d51-b76c-5fd759a5e647/volumes" Jul 15 20:16:32.431098 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:32.431011 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-f84zs"] Jul 15 20:16:40.637616 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:40.637580 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl"] Jul 15 20:16:40.638064 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:40.637794 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" containerID="cri-o://1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919" gracePeriod=30 Jul 15 20:16:45.894150 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.894125 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:16:45.999185 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999152 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-envoy\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999200 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-token\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999253 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-socket\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999273 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-podinfo\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999290 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istiod-ca-cert\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999304 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-credential-socket\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999329 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-data\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999392 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999352 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-certs\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999761 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999403 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssgcp\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-kube-api-access-ssgcp\") pod \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\" (UID: \"764dcf5b-53b7-440a-8d68-a9d421ab31e3\") " Jul 15 20:16:45.999761 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999589 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:45.999761 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999715 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:45.999761 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999735 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:16:46.000012 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:45.999936 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:46.000502 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.000471 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:46.000797 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.000774 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:46.001620 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.001587 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:16:46.001741 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.001677 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:16:46.002141 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.002120 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-kube-api-access-ssgcp" (OuterVolumeSpecName: "kube-api-access-ssgcp") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "kube-api-access-ssgcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:16:46.002212 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.002134 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "764dcf5b-53b7-440a-8d68-a9d421ab31e3" (UID: "764dcf5b-53b7-440a-8d68-a9d421ab31e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:16:46.101175 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101123 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-podinfo\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.101175 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101169 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istiod-ca-cert\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.101175 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101178 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-credential-socket\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.101175 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101187 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-data\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.101175 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101195 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-workload-certs\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.101484 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101203 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssgcp\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-kube-api-access-ssgcp\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.101484 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101211 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-envoy\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.101484 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.101221 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764dcf5b-53b7-440a-8d68-a9d421ab31e3-istio-token\") on node \"ip-10-0-142-40.ec2.internal\" DevicePath \"\"" Jul 15 20:16:46.697506 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.697467 2573 generic.go:356] "Generic (PLEG): container finished" podID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerID="1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919" exitCode=0 Jul 15 20:16:46.697674 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.697532 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" Jul 15 20:16:46.697674 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.697552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" event={"ID":"764dcf5b-53b7-440a-8d68-a9d421ab31e3","Type":"ContainerDied","Data":"1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919"} Jul 15 20:16:46.697674 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.697624 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl" event={"ID":"764dcf5b-53b7-440a-8d68-a9d421ab31e3","Type":"ContainerDied","Data":"aa263302303ff801b463c03a120bd01572a93e504ab4a8238243463d4cf22fce"} Jul 15 20:16:46.697674 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.697659 2573 scope.go:117] "RemoveContainer" containerID="1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919" Jul 15 20:16:46.707851 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.707832 2573 scope.go:117] "RemoveContainer" containerID="5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253" Jul 15 20:16:46.715910 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.715889 2573 scope.go:117] "RemoveContainer" containerID="1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919" Jul 15 20:16:46.716201 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:16:46.716182 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919\": container with ID starting with 1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919 not found: ID does not exist" containerID="1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919" Jul 15 20:16:46.716248 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.716213 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919"} err="failed to get container status \"1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919\": rpc error: code = NotFound desc = could not find container \"1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919\": container with ID starting with 1a8d9817bcd2563f702baa27d03cfaacf848783c5bb6fb36b5bdcbfef5e63919 not found: ID does not exist" Jul 15 20:16:46.716248 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.716232 2573 scope.go:117] "RemoveContainer" containerID="5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253" Jul 15 20:16:46.716466 ip-10-0-142-40 kubenswrapper[2573]: E0715 20:16:46.716448 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253\": container with ID starting with 5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253 not found: ID does not exist" containerID="5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253" Jul 15 20:16:46.716509 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.716472 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253"} err="failed to get container status \"5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253\": rpc error: code = NotFound desc = could not find container \"5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253\": container with ID starting with 5a8d76d26c0087879125d324c456fe5739d59f8286a58cdc141adc212e978253 not found: ID does not exist" Jul 15 20:16:46.719302 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.719281 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl"] Jul 15 20:16:46.723429 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:46.723408 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b72424-openshift-default-6c85c69b4c-s8ngl"] Jul 15 20:16:47.130771 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:47.130695 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4kfsg_b68bcc89-e760-4c37-a111-aa9c885dedbd/manager/0.log" Jul 15 20:16:47.248233 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:47.248206 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-28qt6_c65b3e58-47cf-4ded-87bd-d90e419b4d92/manager/0.log" Jul 15 20:16:47.496064 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:47.496002 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7zhk8_f1997ede-bf93-4836-b5e4-73337a6d1113/server/0.log" Jul 15 20:16:47.616432 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:47.616403 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lnbsw_277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865/manager/0.log" Jul 15 20:16:47.737458 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:47.737434 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58dd997468-klp69_6220d68c-c0f6-4b13-b355-3efa1093638c/copy-manifests/0.log" Jul 15 20:16:47.743606 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:47.743584 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58dd997468-klp69_6220d68c-c0f6-4b13-b355-3efa1093638c/manager/0.log" Jul 15 20:16:48.695993 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:48.695962 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" path="/var/lib/kubelet/pods/764dcf5b-53b7-440a-8d68-a9d421ab31e3/volumes" Jul 15 20:16:51.914936 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:51.914898 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cnxc2_b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8/manager/0.log" Jul 15 20:16:52.027169 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:52.027147 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-fcwlm_61fd86f6-ba8a-4893-8773-315fe9d1f8d6/kuadrant-console-plugin/0.log" Jul 15 20:16:52.150105 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:52.150071 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2pbn2_1332a400-9cdd-4e7c-b86c-f9bda5e05bd4/registry-server/0.log" Jul 15 20:16:52.276294 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:52.276216 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-smhhn_ac911a83-e947-49b8-a4ef-359dea0e4f5b/manager/0.log" Jul 15 20:16:52.395473 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:52.395445 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-f84zs_df252a02-9807-424b-b688-214d8671578f/limitador/0.log" Jul 15 20:16:52.523817 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:52.523787 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4qv9t_403701c4-e563-4ed1-b4a3-0f10a99b4cf4/manager/0.log" Jul 15 20:16:52.900330 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:52.900294 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6789c8m2_11b2785f-dd7a-4ba9-a256-f54fb00ce82a/istio-proxy/0.log" Jul 15 20:16:53.269704 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:53.269675 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b66bc9f5-wz2w9_0378c7a5-591a-45ee-aa34-2d7336d7e01d/kube-auth-proxy/0.log" Jul 15 20:16:53.397918 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:53.397889 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vc7zb_2b4cc9d4-1697-4718-9841-56e6fd40bde8/istio-proxy/1.log" Jul 15 20:16:53.663951 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:53.663926 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-j8smn_814885eb-a332-4a7c-a701-7950cdc3e6ba/payload-processing/0.log" Jul 15 20:16:54.420317 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:54.420285 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5sgr5_06c84d29-fb69-4072-a137-82a2b3dd493d/main/0.log" Jul 15 20:16:54.798718 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:16:54.798628 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-sg7rt_dcc6aaef-d7af-42f0-8410-0a65c03750c3/main/0.log" Jul 15 20:17:04.496816 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:04.496768 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f9fvm_df75c58a-3f8a-4b70-8a59-a768bb296b53/global-pull-secret-syncer/0.log" Jul 15 20:17:04.604908 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:04.604883 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x5b2v_0f36152a-ffe5-4e8d-b335-d52517232ba1/konnectivity-agent/0.log" Jul 15 20:17:04.671131 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:04.671089 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-40.ec2.internal_eb0fb17f4c4321230ae8f9ba72a0414e/haproxy/0.log" Jul 15 20:17:09.874012 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:09.873984 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cnxc2_b3dcacd9-cbb2-4a7d-b4be-ce77844b9aa8/manager/0.log" Jul 15 20:17:09.924253 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:09.924230 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-fcwlm_61fd86f6-ba8a-4893-8773-315fe9d1f8d6/kuadrant-console-plugin/0.log" Jul 15 20:17:09.962050 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:09.962016 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-2pbn2_1332a400-9cdd-4e7c-b86c-f9bda5e05bd4/registry-server/0.log" Jul 15 20:17:10.081478 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:10.081435 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-smhhn_ac911a83-e947-49b8-a4ef-359dea0e4f5b/manager/0.log" Jul 15 20:17:10.117133 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:10.117105 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-f84zs_df252a02-9807-424b-b688-214d8671578f/limitador/0.log" Jul 15 20:17:10.181361 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:10.181327 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4qv9t_403701c4-e563-4ed1-b4a3-0f10a99b4cf4/manager/0.log" Jul 15 20:17:12.450276 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:12.450244 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvcnz_42da4001-b896-4ef5-a791-840a417ebcd5/node-exporter/0.log" Jul 15 20:17:12.474734 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:12.474695 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvcnz_42da4001-b896-4ef5-a791-840a417ebcd5/kube-rbac-proxy/0.log" Jul 15 20:17:12.498265 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:12.498241 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvcnz_42da4001-b896-4ef5-a791-840a417ebcd5/init-textfile/0.log" Jul 15 20:17:14.348985 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.348942 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-95x5f_a4df817f-0d0b-4c52-8f92-d12a27c50869/networking-console-plugin/0.log" Jul 15 20:17:14.954591 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.954549 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh"] Jul 15 20:17:14.955024 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955006 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955027 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955047 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5110e51f-2488-4414-a065-494d04ee22c4" containerName="main" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955055 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5110e51f-2488-4414-a065-494d04ee22c4" containerName="main" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955070 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e33499e-e231-4413-885c-2a95eb162eed" containerName="main" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955078 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e33499e-e231-4413-885c-2a95eb162eed" containerName="main" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955088 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955096 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" Jul 15 20:17:14.955136 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955132 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" Jul 15 20:17:14.955543 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955141 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" Jul 15 20:17:14.955543 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955233 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e33499e-e231-4413-885c-2a95eb162eed" containerName="main" Jul 15 20:17:14.955543 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955251 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="764dcf5b-53b7-440a-8d68-a9d421ab31e3" containerName="istio-proxy" Jul 15 20:17:14.955543 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955260 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5110e51f-2488-4414-a065-494d04ee22c4" containerName="main" Jul 15 20:17:14.955543 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955271 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" Jul 15 20:17:14.955543 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.955281 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a5cf3d-0db5-4d51-b76c-5fd759a5e647" containerName="istio-proxy" Jul 15 20:17:14.958981 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.958958 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:14.959566 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.959537 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/1.log" Jul 15 20:17:14.961011 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.960990 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7q79m\"/\"openshift-service-ca.crt\"" Jul 15 20:17:14.961719 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.961697 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7q79m\"/\"default-dockercfg-ptx96\"" Jul 15 20:17:14.961884 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.961720 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7q79m\"/\"kube-root-ca.crt\"" Jul 15 20:17:14.967605 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.967587 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mt85z_c60bcad4-3d75-45aa-a076-1c84891736f5/console-operator/2.log" Jul 15 20:17:14.968668 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:14.968650 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh"] Jul 15 20:17:15.054298 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.054264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-sys\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.054477 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.054321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-proc\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.054477 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.054368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-lib-modules\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.054477 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.054408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dj82\" (UniqueName: \"kubernetes.io/projected/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-kube-api-access-9dj82\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.054477 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.054477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-podres\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155442 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dj82\" (UniqueName: \"kubernetes.io/projected/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-kube-api-access-9dj82\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155618 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-podres\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155618 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-sys\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155618 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-proc\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155618 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-podres\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155618 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-lib-modules\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155858 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-proc\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155858 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155653 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-sys\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.155858 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.155716 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-lib-modules\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.163731 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.163707 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dj82\" (UniqueName: \"kubernetes.io/projected/657a3d99-772f-4ae4-9a32-d2bf07c49ca7-kube-api-access-9dj82\") pod \"perf-node-gather-daemonset-2zlhh\" (UID: \"657a3d99-772f-4ae4-9a32-d2bf07c49ca7\") " pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.270785 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.270689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.404918 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.404891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh"] Jul 15 20:17:15.406661 ip-10-0-142-40 kubenswrapper[2573]: W0715 20:17:15.406622 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod657a3d99_772f_4ae4_9a32_d2bf07c49ca7.slice/crio-ed3551599b95e53b189b5cbefd16596abc1a40ec4cc044d38356012c3c67eacb WatchSource:0}: Error finding container ed3551599b95e53b189b5cbefd16596abc1a40ec4cc044d38356012c3c67eacb: Status 404 returned error can't find the container with id ed3551599b95e53b189b5cbefd16596abc1a40ec4cc044d38356012c3c67eacb Jul 15 20:17:15.838707 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.838670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" event={"ID":"657a3d99-772f-4ae4-9a32-d2bf07c49ca7","Type":"ContainerStarted","Data":"b850348121daee5e15ede25e1c896f2bfed572433667886a3bdaf80a0a00521f"} Jul 15 20:17:15.838939 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.838765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" event={"ID":"657a3d99-772f-4ae4-9a32-d2bf07c49ca7","Type":"ContainerStarted","Data":"ed3551599b95e53b189b5cbefd16596abc1a40ec4cc044d38356012c3c67eacb"} Jul 15 20:17:15.838939 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.838788 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:15.854057 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:15.854006 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" podStartSLOduration=1.8539904329999999 podStartE2EDuration="1.853990433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:17:15.853195459 +0000 UTC m=+2917.714399813" watchObservedRunningTime="2026-07-15 20:17:15.853990433 +0000 UTC m=+2917.715194785" Jul 15 20:17:16.128123 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:16.128039 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-8rzjv_2ed6d1d4-eb46-4b34-b127-f5737e8a8c1b/volume-data-source-validator/0.log" Jul 15 20:17:17.141562 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:17.141537 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4gz28_8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6/dns/0.log" Jul 15 20:17:17.162647 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:17.162622 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4gz28_8bc122d6-cbc3-4c34-99f3-9c420f4f8ae6/kube-rbac-proxy/0.log" Jul 15 20:17:17.280921 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:17.280894 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7l44b_2514b799-dd92-4c5d-9a7e-63f79cab71ef/dns-node-resolver/0.log" Jul 15 20:17:17.918706 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:17.918675 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzq5g_cc2dcea3-f197-4b33-b77f-00716c3c698c/node-ca/0.log" Jul 15 20:17:18.815490 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:18.815464 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6789c8m2_11b2785f-dd7a-4ba9-a256-f54fb00ce82a/istio-proxy/0.log" Jul 15 20:17:18.994363 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:18.994333 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b66bc9f5-wz2w9_0378c7a5-591a-45ee-aa34-2d7336d7e01d/kube-auth-proxy/0.log" Jul 15 20:17:19.023580 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:19.023556 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vc7zb_2b4cc9d4-1697-4718-9841-56e6fd40bde8/istio-proxy/0.log" Jul 15 20:17:19.052426 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:19.052394 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vc7zb_2b4cc9d4-1697-4718-9841-56e6fd40bde8/istio-proxy/1.log" Jul 15 20:17:19.108681 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:19.108602 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-j8smn_814885eb-a332-4a7c-a701-7950cdc3e6ba/payload-processing/0.log" Jul 15 20:17:19.834954 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:19.834926 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-72d5n_bed9667d-b54b-4056-aa7b-300f7905aa67/serve-healthcheck-canary/0.log" Jul 15 20:17:20.333616 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:20.333579 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5w487_ca6fa370-fa41-45c9-8b77-d0eeb61379ec/kube-rbac-proxy/0.log" Jul 15 20:17:20.354312 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:20.354288 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5w487_ca6fa370-fa41-45c9-8b77-d0eeb61379ec/exporter/0.log" Jul 15 20:17:20.376039 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:20.376019 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5w487_ca6fa370-fa41-45c9-8b77-d0eeb61379ec/extractor/0.log" Jul 15 20:17:21.855035 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:21.855008 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7q79m/perf-node-gather-daemonset-2zlhh" Jul 15 20:17:22.581593 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:22.581560 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4kfsg_b68bcc89-e760-4c37-a111-aa9c885dedbd/manager/0.log" Jul 15 20:17:22.760181 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:22.760145 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-28qt6_c65b3e58-47cf-4ded-87bd-d90e419b4d92/manager/0.log" Jul 15 20:17:22.934732 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:22.934703 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7zhk8_f1997ede-bf93-4836-b5e4-73337a6d1113/server/0.log" Jul 15 20:17:23.000635 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:23.000605 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lnbsw_277e1c02-9dc1-4e0f-8e01-d4e4e2fbe865/manager/0.log" Jul 15 20:17:23.028454 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:23.028423 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58dd997468-klp69_6220d68c-c0f6-4b13-b355-3efa1093638c/manager/0.log" Jul 15 20:17:23.048955 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:23.048927 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58dd997468-klp69_6220d68c-c0f6-4b13-b355-3efa1093638c/copy-manifests/0.log" Jul 15 20:17:24.629343 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:24.629310 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-x5f6r_f2ddbf45-ea28-46ec-9622-6df5e2e15771/openshift-lws-operator/0.log" Jul 15 20:17:25.524770 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:25.524743 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6789c8m2_11b2785f-dd7a-4ba9-a256-f54fb00ce82a/istio-proxy/0.log" Jul 15 20:17:25.673863 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:25.673778 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b66bc9f5-wz2w9_0378c7a5-591a-45ee-aa34-2d7336d7e01d/kube-auth-proxy/0.log" Jul 15 20:17:25.697844 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:25.697798 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vc7zb_2b4cc9d4-1697-4718-9841-56e6fd40bde8/istio-proxy/0.log" Jul 15 20:17:25.727765 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:25.727738 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vc7zb_2b4cc9d4-1697-4718-9841-56e6fd40bde8/istio-proxy/1.log" Jul 15 20:17:25.764279 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:25.764254 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-j8smn_814885eb-a332-4a7c-a701-7950cdc3e6ba/payload-processing/0.log" Jul 15 20:17:27.062165 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:27.062077 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6789c8m2_11b2785f-dd7a-4ba9-a256-f54fb00ce82a/istio-proxy/0.log" Jul 15 20:17:27.202763 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:27.202727 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b66bc9f5-wz2w9_0378c7a5-591a-45ee-aa34-2d7336d7e01d/kube-auth-proxy/0.log" Jul 15 20:17:27.223614 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:27.223592 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vc7zb_2b4cc9d4-1697-4718-9841-56e6fd40bde8/istio-proxy/0.log" Jul 15 20:17:27.252527 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:27.252499 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-vc7zb_2b4cc9d4-1697-4718-9841-56e6fd40bde8/istio-proxy/1.log" Jul 15 20:17:27.296197 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:27.296167 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-j8smn_814885eb-a332-4a7c-a701-7950cdc3e6ba/payload-processing/0.log" Jul 15 20:17:29.684404 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:29.684379 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xgkxk_07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1/migrator/0.log" Jul 15 20:17:29.705084 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:29.705060 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xgkxk_07a2f8ca-1c14-465d-a6d7-2a1ad7b228f1/graceful-termination/0.log" Jul 15 20:17:31.207679 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.207637 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfr4h_48b7ce9d-c732-4b21-a7ef-bd8f9963a183/kube-multus-additional-cni-plugins/0.log" Jul 15 20:17:31.228288 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.228265 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfr4h_48b7ce9d-c732-4b21-a7ef-bd8f9963a183/egress-router-binary-copy/0.log" Jul 15 20:17:31.250070 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.250048 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfr4h_48b7ce9d-c732-4b21-a7ef-bd8f9963a183/cni-plugins/0.log" Jul 15 20:17:31.271955 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.271929 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfr4h_48b7ce9d-c732-4b21-a7ef-bd8f9963a183/bond-cni-plugin/0.log" Jul 15 20:17:31.292852 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.292828 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfr4h_48b7ce9d-c732-4b21-a7ef-bd8f9963a183/routeoverride-cni/0.log" Jul 15 20:17:31.312670 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.312602 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfr4h_48b7ce9d-c732-4b21-a7ef-bd8f9963a183/whereabouts-cni-bincopy/0.log" Jul 15 20:17:31.332743 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.332725 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gfr4h_48b7ce9d-c732-4b21-a7ef-bd8f9963a183/whereabouts-cni/0.log" Jul 15 20:17:31.536566 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.536535 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tlph8_00ec9d85-c523-457e-8688-a96deb7d1ed7/kube-multus/0.log" Jul 15 20:17:31.640905 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.640823 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j5zpl_7ab526b6-f5f9-4a7c-ab2b-04eb387d9520/network-metrics-daemon/0.log" Jul 15 20:17:31.659080 ip-10-0-142-40 kubenswrapper[2573]: I0715 20:17:31.659057 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j5zpl_7ab526b6-f5f9-4a7c-ab2b-04eb387d9520/kube-rbac-proxy/0.log"