Jul 22 00:24:52.713444 ip-10-0-131-63 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 00:24:52.713455 ip-10-0-131-63 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 00:24:52.713463 ip-10-0-131-63 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 00:24:52.713766 ip-10-0-131-63 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 00:25:02.945610 ip-10-0-131-63 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 00:25:02.945629 ip-10-0-131-63 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6a98dfb745204728bf3eb36ee8b16eab -- Jul 22 00:26:55.255271 ip-10-0-131-63 systemd[1]: Starting Kubernetes Kubelet... Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.656380 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663177 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663194 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663198 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663201 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663205 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663209 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663213 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:55.689961 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663216 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663219 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663221 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663224 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663227 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663231 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663234 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663237 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663246 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663249 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663252 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663255 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663259 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663264 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663268 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663271 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663274 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663277 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663279 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:55.690988 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663282 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663284 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663287 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663290 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663292 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663295 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663298 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663300 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663303 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663306 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663308 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663311 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663314 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663316 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663319 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663322 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663324 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663334 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663339 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:55.691574 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663342 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663346 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663349 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663353 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663355 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663358 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663361 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663363 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663366 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663369 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663372 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663375 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663379 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663381 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663384 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663386 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663389 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663392 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663394 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:55.692127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663396 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663399 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663402 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663404 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663407 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663410 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663413 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663415 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663418 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663420 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663423 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663425 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663428 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663431 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663434 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663437 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663440 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663444 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663447 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663450 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:55.692634 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663452 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663455 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663457 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663460 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663462 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663465 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663467 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663470 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663473 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663476 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663480 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663483 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663486 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663489 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.663492 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664231 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664238 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664242 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664245 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664248 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:55.693356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664251 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664254 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664258 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664261 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664263 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664267 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664270 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664273 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664276 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664279 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664284 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664288 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664291 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664294 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664297 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664300 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664304 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664306 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664309 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664312 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:55.693941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664314 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664317 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664319 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664322 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664324 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664328 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664330 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664333 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664336 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664339 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664342 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664345 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664348 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664350 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664353 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664356 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664358 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664362 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664365 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664368 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:55.694593 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664370 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664373 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664376 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664379 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664381 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664384 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664387 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664390 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664392 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664395 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664398 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664404 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664406 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664411 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664414 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664417 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664420 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664423 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664426 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:55.695160 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664429 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664432 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664435 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664441 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664444 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664447 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664450 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664453 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664456 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664458 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664461 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664464 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664466 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664469 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664472 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664474 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664477 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664481 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664484 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664486 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:55.695766 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664489 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664492 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664494 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664496 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664499 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664503 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664507 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664510 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664513 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664515 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664518 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664520 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664523 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664525 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664528 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664932 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664937 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664940 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664943 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664946 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:55.696305 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664949 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664951 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664954 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664957 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664959 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664962 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664964 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664967 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664970 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664981 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664984 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664987 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664990 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664993 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.664995 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665000 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665003 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665006 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665009 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:55.697101 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665012 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665015 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665019 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665021 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665025 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665028 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665030 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665033 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665038 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665040 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665043 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665046 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665049 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665051 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665054 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665056 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665059 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665061 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665064 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665066 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:55.697780 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665069 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665073 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665076 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665079 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665081 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665086 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665088 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665091 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665094 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665097 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665099 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665102 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665104 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665107 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665110 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665112 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665115 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665117 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665120 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665122 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665126 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:55.698316 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665129 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665132 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665134 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665137 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665141 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665144 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665147 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665150 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665153 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665156 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665159 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665162 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665165 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665168 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665171 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665174 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665179 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665182 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665184 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:55.698903 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665187 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665190 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665193 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665195 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665199 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665201 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665204 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665207 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665209 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665212 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665215 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665217 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665221 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665224 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665228 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665303 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665311 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665318 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665323 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665327 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 00:26:55.699414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665331 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665335 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665339 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665343 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665346 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665350 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665353 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665357 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665360 2562 flags.go:64] FLAG: --cgroup-root="" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665365 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665369 2562 flags.go:64] FLAG: --client-ca-file="" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665372 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665375 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665379 2562 flags.go:64] FLAG: --cluster-domain="" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665382 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665386 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665389 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665392 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665396 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665400 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665403 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665406 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665409 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665412 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 00:26:55.699970 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665417 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665420 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665423 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665427 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665430 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665433 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665436 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665439 2562 flags.go:64] FLAG: --enable-server="true" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665442 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665447 2562 flags.go:64] FLAG: --event-burst="100" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665450 2562 flags.go:64] FLAG: --event-qps="50" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665453 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665456 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665459 2562 flags.go:64] FLAG: --eviction-hard="" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665463 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665466 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665469 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665474 2562 flags.go:64] FLAG: --eviction-soft="" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665478 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665481 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665484 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665487 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665490 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665493 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665496 2562 flags.go:64] FLAG: --feature-gates="" Jul 22 00:26:55.700566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665500 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665503 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665506 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665509 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665512 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665515 2562 flags.go:64] FLAG: --help="false" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665518 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-131-63.ec2.internal" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665525 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665528 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665531 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665535 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665538 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665541 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665544 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665547 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665550 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665553 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665556 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665559 2562 flags.go:64] FLAG: --kube-reserved="" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665562 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665565 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665568 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665570 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665573 2562 flags.go:64] FLAG: --lock-file="" Jul 22 00:26:55.701682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665578 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665581 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665585 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665590 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665594 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665597 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665599 2562 flags.go:64] FLAG: --logging-format="text" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665602 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665606 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665608 2562 flags.go:64] FLAG: --manifest-url="" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665611 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665620 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665623 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665627 2562 flags.go:64] FLAG: --max-pods="110" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665630 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665634 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665638 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665662 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665665 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665668 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665671 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665679 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665683 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665686 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 00:26:55.702949 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665689 2562 flags.go:64] FLAG: --pod-cidr="" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665692 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665700 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665703 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665707 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665710 2562 flags.go:64] FLAG: --port="10250" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665713 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665716 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05cff368d403343f8" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665721 2562 flags.go:64] FLAG: --qos-reserved="" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665724 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665727 2562 flags.go:64] FLAG: --register-node="true" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665731 2562 flags.go:64] FLAG: --register-with-taints="" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665735 2562 flags.go:64] FLAG: --registry-burst="10" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665738 2562 flags.go:64] FLAG: --registry-qps="5" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665741 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665744 2562 flags.go:64] FLAG: --reserved-memory="" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665748 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665751 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665754 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665757 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665760 2562 flags.go:64] FLAG: --runonce="false" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665763 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665766 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665769 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665774 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 00:26:55.704016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665777 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665780 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665784 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665787 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665789 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665793 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665795 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665798 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665802 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665805 2562 flags.go:64] FLAG: --system-cgroups="" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665807 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665813 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665816 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665819 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665823 2562 flags.go:64] FLAG: --tls-min-version="" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665828 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665831 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665834 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665837 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665841 2562 flags.go:64] FLAG: --v="2" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665846 2562 flags.go:64] FLAG: --version="false" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665849 2562 flags.go:64] FLAG: --vmodule="" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665853 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.665857 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665971 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:55.705941 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665975 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665978 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665981 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665983 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665986 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665989 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665994 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665996 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.665999 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666002 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666004 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666007 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666009 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666013 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666015 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666018 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666021 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666024 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666026 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666029 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:55.706994 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666032 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666034 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666038 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666041 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666043 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666046 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666049 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666052 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666055 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666057 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666060 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666063 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666065 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666069 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666073 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666076 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666078 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666081 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666085 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:55.831802 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666088 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:55.738623 ip-10-0-131-63 systemd[1]: Started Kubernetes Kubelet. Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666090 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666093 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666096 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666099 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666101 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666104 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666107 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666109 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666112 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666114 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666117 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666119 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666122 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666125 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666128 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666131 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666133 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666136 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:55.833083 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666139 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666141 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666144 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666147 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666150 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666152 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666156 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666159 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666162 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666165 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666168 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666171 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666174 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666178 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666182 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666185 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666188 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666190 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666193 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:55.834556 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666196 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666198 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666201 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666204 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666207 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666209 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666212 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666215 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666217 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666221 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666223 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666226 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666228 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666231 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666234 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666237 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666240 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666242 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666245 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666248 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:55.835600 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.666251 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.666257 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.678584 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.678609 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.678627 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.678633 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678724 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678730 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678734 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678738 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678741 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678744 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678747 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678750 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:55.836686 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678753 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678756 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678758 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678762 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678764 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678767 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678770 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678772 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678777 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678780 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678783 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678791 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678794 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678797 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678799 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678802 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678804 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678807 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678809 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:55.837385 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678812 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678815 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678821 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678824 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678827 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678829 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678832 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678834 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678837 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678840 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678843 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678846 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678848 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678851 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678854 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678857 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678859 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678862 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678865 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678868 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:55.838400 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678871 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678873 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678876 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678882 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678886 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678889 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678892 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678894 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678897 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678900 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678902 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678905 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678908 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678910 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678914 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678917 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678919 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678922 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678925 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678928 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:55.840214 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678931 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678933 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678936 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678938 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678941 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678943 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678946 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678949 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678952 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678954 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678957 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678959 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678962 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678965 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678968 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678972 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678974 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678977 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678979 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678982 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:55.841259 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678985 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678987 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678990 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678993 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.678996 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679000 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679004 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679007 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679010 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679013 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679016 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679018 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.679023 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679140 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:55.842428 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679145 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679149 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679151 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679154 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679158 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679160 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679163 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679166 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679169 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679172 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679174 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679178 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679180 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679184 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679187 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679189 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679192 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679195 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679198 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679200 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:55.843210 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679203 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679205 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679208 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679211 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679214 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679217 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679220 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679222 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679226 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679228 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679231 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679233 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679236 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679238 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679241 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679243 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679246 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679248 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679251 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679254 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:55.844383 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679256 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679259 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679261 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679265 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679269 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679273 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679276 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679278 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679281 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679283 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679286 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679290 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679293 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679296 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679299 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679302 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679305 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679309 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679312 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679314 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:55.845321 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679317 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679320 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679323 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679325 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679327 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679330 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679333 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679335 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679337 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679340 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679342 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679345 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679347 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679350 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679352 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679355 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679358 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679362 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679365 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679367 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:55.848231 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679370 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679372 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679375 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679378 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679380 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679383 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679385 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679388 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679390 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679394 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679397 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679399 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679402 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679405 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679409 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679412 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679415 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:55.864234 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:55.679418 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.679423 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.679546 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.681636 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.683058 2562 server.go:1019] "Starting client certificate rotation" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.683154 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.683197 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.685261 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.689962 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.700793 2562 log.go:25] "Validated CRI v1 runtime API" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.703722 2562 log.go:25] "Validated CRI v1 image API" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.705479 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 00:26:55.864903 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.708491 2562 fs.go:135] Filesystem UUIDs: map[2536010b-af22-4764-b5fb-812cb71db99d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d9de9ece-8b72-496b-bdda-bf97b2a551e2:/dev/nvme0n1p4] Jul 22 00:26:55.865268 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.708507 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 00:26:55.865268 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.713347 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.716627 2562 manager.go:216] Machine: {Timestamp:2026-07-22 00:26:55.716144416 +0000 UTC m=+0.346376948 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199326 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd26a8c8c12923b1e6d9460bdb7b6 SystemUUID:ec2fd26a-8c8c-1292-3b1e-6d9460bdb7b6 BootID:6a98dfb7-4520-4728-bf3e-b36ee8b16eab Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b0:4d:5e:02:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b0:4d:5e:02:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:e6:a4:c7:ef:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.716755 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.716885 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.717983 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.718007 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-63.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.718727 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.718737 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.718744 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.719349 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.720836 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.724360 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.724491 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.724522 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.724532 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.724544 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.724553 2562 kubelet.go:398] "Adding apiserver pod source" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.724563 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 00:26:55.865480 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.725051 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.725072 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.727856 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.729662 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.729689 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731143 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731159 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731165 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731171 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731180 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731186 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731194 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731199 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731207 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731212 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731222 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731228 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731748 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.731764 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.737702 2562 server.go:1268] "Started kubelet" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.737791 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gv9gg" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.737848 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.737843 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.737960 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.738464 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.738727 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.738788 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-63.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.739731 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.739989 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.742481 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.744077 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-63.ec2.internal.18c47504c7c84121 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-63.ec2.internal,UID:ip-10-0-131-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-63.ec2.internal,},FirstTimestamp:2026-07-22 00:26:55.737667873 +0000 UTC m=+0.367900407,LastTimestamp:2026-07-22 00:26:55.737667873 +0000 UTC m=+0.367900407,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-63.ec2.internal,}" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.746142 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.746674 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.746779 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747143 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747159 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747179 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747325 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747335 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747337 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gv9gg" Jul 22 00:26:55.866049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747867 2562 factory.go:55] Registering systemd factory Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.747888 2562 factory.go:223] Registration of the systemd container factory successfully Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.748148 2562 factory.go:153] Registering CRI-O factory Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.748338 2562 factory.go:223] Registration of the crio container factory successfully Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.748400 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.748425 2562 factory.go:103] Registering Raw factory Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.748441 2562 manager.go:1196] Started watching for new ooms in manager Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.757531 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.757663 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.759491 2562 manager.go:349] Starting recovery of all containers Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.769681 2562 manager.go:354] Recovery completed Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.774176 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.776117 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.776145 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.776158 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.776653 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.776664 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.776680 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.776688 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.777366 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.777389 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.777400 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.777732 2562 policy_none.go:49] "None policy: Start" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.777749 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.777758 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.778250 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-63.ec2.internal.18c47504ca13246c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-63.ec2.internal,UID:ip-10-0-131-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-63.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-63.ec2.internal,},FirstTimestamp:2026-07-22 00:26:55.776130156 +0000 UTC m=+0.406362687,LastTimestamp:2026-07-22 00:26:55.776130156 +0000 UTC m=+0.406362687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-63.ec2.internal,}" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.778728 2562 policy_none.go:47] "Start" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.824254 2562 manager.go:337] "Starting Device Plugin manager" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.824475 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.824486 2562 server.go:87] "Starting device plugin registration server" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.824734 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.824744 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.824853 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.824951 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.824961 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.825503 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 00:26:55.867078 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.825546 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.898042 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.899111 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.899142 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.899164 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.899196 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.901212 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.925741 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.929754 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.929782 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.929792 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.929813 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:55.937071 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.937090 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-63.ec2.internal\": node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:55.957703 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.000027 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal"] Jul 22 00:26:56.000124 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.000123 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:56.001982 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.001528 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:56.001982 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.001560 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:56.001982 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.001575 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.003743 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.003916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.003940 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.004480 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.004500 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.004512 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.004484 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.004564 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.004578 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.006762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.006787 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.007413 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.007440 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.007455 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.047706 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-63.ec2.internal\" not found" node="ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.048784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f510730f581f273c10f59e1671cc03f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"f510730f581f273c10f59e1671cc03f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.048805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f510730f581f273c10f59e1671cc03f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"f510730f581f273c10f59e1671cc03f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.048823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cee79eddbd8dbbcb832bb8dad2fba384-config\") pod \"kube-apiserver-proxy-ip-10-0-131-63.ec2.internal\" (UID: \"cee79eddbd8dbbcb832bb8dad2fba384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.050913 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-63.ec2.internal\" not found" node="ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.129370 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.058292 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.185733 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.149256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cee79eddbd8dbbcb832bb8dad2fba384-config\") pod \"kube-apiserver-proxy-ip-10-0-131-63.ec2.internal\" (UID: \"cee79eddbd8dbbcb832bb8dad2fba384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.185733 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.149292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f510730f581f273c10f59e1671cc03f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"f510730f581f273c10f59e1671cc03f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.185733 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.149310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f510730f581f273c10f59e1671cc03f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"f510730f581f273c10f59e1671cc03f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.185733 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.149350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cee79eddbd8dbbcb832bb8dad2fba384-config\") pod \"kube-apiserver-proxy-ip-10-0-131-63.ec2.internal\" (UID: \"cee79eddbd8dbbcb832bb8dad2fba384\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.185733 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.149357 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f510730f581f273c10f59e1671cc03f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"f510730f581f273c10f59e1671cc03f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.185733 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.149352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f510730f581f273c10f59e1671cc03f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"f510730f581f273c10f59e1671cc03f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.185733 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.159367 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.348404 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.260003 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.350656 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.350350 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.444235 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.353435 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 22 00:26:56.444235 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.361071 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.529113 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.461487 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.674478 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.561795 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.674478 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.662559 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.682712 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.682690 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 00:26:56.682862 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.682843 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 00:26:56.761221 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.761182 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 00:21:55 +0000 UTC" deadline="2028-03-09 16:16:11.933990887 +0000 UTC" Jul 22 00:26:56.761221 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.761215 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14319h49m15.172778499s" Jul 22 00:26:56.763233 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.763215 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.791158 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.791132 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:56.791937 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.791919 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:56.864124 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.864093 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:56.930627 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:56.930562 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf510730f581f273c10f59e1671cc03f4.slice/crio-824f674f9204951091e5833e0e2b28b1ee7e9c444f645e8d77b14b009571809c WatchSource:0}: Error finding container 824f674f9204951091e5833e0e2b28b1ee7e9c444f645e8d77b14b009571809c: Status 404 returned error can't find the container with id 824f674f9204951091e5833e0e2b28b1ee7e9c444f645e8d77b14b009571809c Jul 22 00:26:56.934856 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:56.934842 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:26:56.955356 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:56.955332 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee79eddbd8dbbcb832bb8dad2fba384.slice/crio-7540234d144ff201b89a9f82f5eae01a91e3d7d5d4655c6c34b8cf107c061d24 WatchSource:0}: Error finding container 7540234d144ff201b89a9f82f5eae01a91e3d7d5d4655c6c34b8cf107c061d24: Status 404 returned error can't find the container with id 7540234d144ff201b89a9f82f5eae01a91e3d7d5d4655c6c34b8cf107c061d24 Jul 22 00:26:56.964672 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:56.964633 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 22 00:26:57.023791 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.023768 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:57.047306 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.047275 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 22 00:26:57.055490 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.055472 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 00:26:57.056277 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.056265 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 22 00:26:57.063662 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.063628 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 00:26:57.725212 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.725180 2562 apiserver.go:52] "Watching apiserver" Jul 22 00:26:57.732761 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.732741 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 00:26:57.734312 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.734290 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-x9f5n","openshift-network-operator/iptables-alerter-27ttx","openshift-ovn-kubernetes/ovnkube-node-zsrgc","kube-system/global-pull-secret-syncer-dzkpd","kube-system/konnectivity-agent-x9ndd","kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal","openshift-dns/node-resolver-jdphr","openshift-image-registry/node-ca-mkzlt","openshift-multus/multus-additional-cni-plugins-mzrpx","openshift-multus/multus-db4kc","openshift-network-diagnostics/network-check-target-8r7bq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx","openshift-cluster-node-tuning-operator/tuned-6dpst","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal"] Jul 22 00:26:57.736509 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.736492 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.738558 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.738540 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.738986 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.738964 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-spqlp\"" Jul 22 00:26:57.739073 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.739007 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 00:26:57.739073 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.739010 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 00:26:57.740323 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.740267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:26:57.740323 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.740300 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 00:26:57.740323 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.740322 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z64sk\"" Jul 22 00:26:57.740997 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.740980 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.743027 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.742890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.743027 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.742944 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:26:57.745513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.744300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 00:26:57.745513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.744421 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 00:26:57.745513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.744729 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wxkc6\"" Jul 22 00:26:57.745513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.744915 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 00:26:57.745513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.745051 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 00:26:57.745513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.745253 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 00:26:57.745513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.745356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 00:26:57.745873 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.745575 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:57.746596 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.746577 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 00:26:57.747729 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.747441 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 00:26:57.747729 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.747468 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9wdn6\"" Jul 22 00:26:57.747729 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.747497 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 00:26:57.747969 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.747952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.749710 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.749693 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 00:26:57.749863 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.749844 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 00:26:57.750122 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.750103 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 00:26:57.750250 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.750207 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sprbv\"" Jul 22 00:26:57.750481 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.750468 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.752118 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.752101 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 00:26:57.753256 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.752820 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thkc2\"" Jul 22 00:26:57.753256 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.753100 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 00:26:57.753712 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.753693 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 00:26:57.754170 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.753956 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 00:26:57.754170 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.754165 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:57.754294 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.754226 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 00:26:57.754294 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.754229 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:26:57.756540 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756424 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e48b5beb-479e-402e-891a-33a0c4c0449c-agent-certs\") pod \"konnectivity-agent-x9ndd\" (UID: \"e48b5beb-479e-402e-891a-33a0c4c0449c\") " pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:57.756540 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-kubernetes\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.756540 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-kubelet\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.756540 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-kubelet-config\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.756540 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e48b5beb-479e-402e-891a-33a0c4c0449c-konnectivity-ca\") pod \"konnectivity-agent-x9ndd\" (UID: \"e48b5beb-479e-402e-891a-33a0c4c0449c\") " pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89tx\" (UniqueName: \"kubernetes.io/projected/71460dc1-1655-4123-b9d8-ded03647803a-kube-api-access-g89tx\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756567 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzkm\" (UniqueName: \"kubernetes.io/projected/441030f4-d8a5-48cf-82d4-4882497d5560-kube-api-access-vdzkm\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-etc-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-node-log\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-dbus\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-modprobe-d\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-host\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-systemd-units\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-ovn\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfkk\" (UniqueName: \"kubernetes.io/projected/2f487ea8-8634-47a0-afc1-22b60a19e52a-kube-api-access-lcfkk\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.756843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysctl-d\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-systemd\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-run\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-tmp\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-systemd\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.756993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-cni-bin\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovnkube-config\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/441030f4-d8a5-48cf-82d4-4882497d5560-hosts-file\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysctl-conf\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmnk\" (UniqueName: \"kubernetes.io/projected/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-kube-api-access-fhmnk\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-var-lib-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovn-node-metrics-cert\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71460dc1-1655-4123-b9d8-ded03647803a-serviceca\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-tuned\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-run-netns\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-log-socket\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.757368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757344 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-cni-netd\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757427 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441030f4-d8a5-48cf-82d4-4882497d5560-tmp-dir\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysconfig\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-sys\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757548 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71460dc1-1655-4123-b9d8-ded03647803a-host\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757586 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-lib-modules\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-var-lib-kubelet\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-slash\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-env-overrides\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.758001 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.757793 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovnkube-script-lib\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.758511 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.758179 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 00:26:57.761321 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.761304 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.763030 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.763001 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hczsn\"" Jul 22 00:26:57.763030 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.763015 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:26:57.763258 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.763210 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 00:26:57.763328 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.763285 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 00:26:57.763776 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.763590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.765431 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.765414 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 00:26:57.765749 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.765711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:26:57.765836 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.765786 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dvpfm\"" Jul 22 00:26:57.765886 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.765830 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:26:57.768357 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.767992 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.769883 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.769865 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 00:26:57.769971 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.769918 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 00:26:57.770025 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.769977 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9nf5t\"" Jul 22 00:26:57.770069 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.770058 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 00:26:57.777405 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.777388 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxx4c" Jul 22 00:26:57.781984 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.781970 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxx4c" Jul 22 00:26:57.848200 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.848173 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 00:26:57.858608 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-lib-modules\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.858608 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-slash\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.858871 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858633 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovnkube-script-lib\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.858871 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858669 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e48b5beb-479e-402e-891a-33a0c4c0449c-agent-certs\") pod \"konnectivity-agent-x9ndd\" (UID: \"e48b5beb-479e-402e-891a-33a0c4c0449c\") " pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:57.858871 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-lib-modules\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.858871 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-slash\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.858871 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-socket-dir-parent\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.858871 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-kubelet-config\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.858871 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89tx\" (UniqueName: \"kubernetes.io/projected/71460dc1-1655-4123-b9d8-ded03647803a-kube-api-access-g89tx\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-system-cni-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858915 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-netns\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-kubelet-config\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-etc-kubernetes\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.858993 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-registration-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-node-log\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-dbus\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-hostroot\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-daemon-config\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-modprobe-d\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-host\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859162 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-node-log\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.859180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-systemd-units\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859214 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-systemd-units\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-dbus\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-modprobe-d\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-host\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-kubelet\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovnkube-script-lib\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-kube-api-access-rrcnn\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-run\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71460dc1-1655-4123-b9d8-ded03647803a-serviceca\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859440 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859454 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-system-cni-dir\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-run\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.859830 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-sys-fs\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-var-lib-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859600 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-k8s-cni-cncf-io\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-var-lib-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-os-release\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-conf-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859737 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-device-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-sys\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71460dc1-1655-4123-b9d8-ded03647803a-host\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71460dc1-1655-4123-b9d8-ded03647803a-serviceca\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpr5w\" (UniqueName: \"kubernetes.io/projected/88f41e66-c8b2-485f-809c-6e8f994c4541-kube-api-access-cpr5w\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859809 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-sys\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71460dc1-1655-4123-b9d8-ded03647803a-host\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmbc\" (UniqueName: \"kubernetes.io/projected/21a83aaa-88e2-4c83-9d39-3e393cef3688-kube-api-access-zhmbc\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.860722 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-host-slash\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-var-lib-kubelet\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-env-overrides\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21a83aaa-88e2-4c83-9d39-3e393cef3688-cni-binary-copy\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859964 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-var-lib-kubelet\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.859990 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-iptables-alerter-script\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-kubernetes\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-kubelet\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e48b5beb-479e-402e-891a-33a0c4c0449c-konnectivity-ca\") pod \"konnectivity-agent-x9ndd\" (UID: \"e48b5beb-479e-402e-891a-33a0c4c0449c\") " pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860079 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-kubernetes\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwmz\" (UniqueName: \"kubernetes.io/projected/b13067b2-8bb6-4faf-8818-e50de72f1db6-kube-api-access-xkwmz\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-cnibin\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzkm\" (UniqueName: \"kubernetes.io/projected/441030f4-d8a5-48cf-82d4-4882497d5560-kube-api-access-vdzkm\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-etc-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-multus-certs\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-ovn\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.861665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860273 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-env-overrides\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfkk\" (UniqueName: \"kubernetes.io/projected/2f487ea8-8634-47a0-afc1-22b60a19e52a-kube-api-access-lcfkk\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysctl-d\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.860362 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-ovn\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860403 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-systemd\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.860436 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret podName:711d23ce-641f-4bf9-bf7a-cce1d84a05b5 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:58.360402761 +0000 UTC m=+2.990635291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret") pod "global-pull-secret-syncer-dzkpd" (UID: "711d23ce-641f-4bf9-bf7a-cce1d84a05b5") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860476 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-kubelet\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-systemd\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-etc-openvswitch\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-tmp\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-systemd\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-cni-bin\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovnkube-config\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e48b5beb-479e-402e-891a-33a0c4c0449c-konnectivity-ca\") pod \"konnectivity-agent-x9ndd\" (UID: \"e48b5beb-479e-402e-891a-33a0c4c0449c\") " pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovn-node-metrics-cert\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-cnibin\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.862603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860684 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-cni-bin\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860736 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-run-systemd\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/441030f4-d8a5-48cf-82d4-4882497d5560-hosts-file\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysctl-conf\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmnk\" (UniqueName: \"kubernetes.io/projected/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-kube-api-access-fhmnk\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysctl-d\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-tuned\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-run-netns\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860945 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/441030f4-d8a5-48cf-82d4-4882497d5560-hosts-file\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-log-socket\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.860994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-cni-netd\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-os-release\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-cni-bin\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.863581 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysctl-conf\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861077 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-cni-multus\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441030f4-d8a5-48cf-82d4-4882497d5560-tmp-dir\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysconfig\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-cni-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-socket-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwgct\" (UniqueName: \"kubernetes.io/projected/8ee4be3b-825a-4893-a581-c0d94530a4cf-kube-api-access-hwgct\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441030f4-d8a5-48cf-82d4-4882497d5560-tmp-dir\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-sysconfig\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861505 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-cni-netd\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-host-run-netns\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861570 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f487ea8-8634-47a0-afc1-22b60a19e52a-log-socket\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.861993 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovnkube-config\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.863217 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e48b5beb-479e-402e-891a-33a0c4c0449c-agent-certs\") pod \"konnectivity-agent-x9ndd\" (UID: \"e48b5beb-479e-402e-891a-33a0c4c0449c\") " pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.863290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-etc-tuned\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.864544 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.863450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-tmp\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.865428 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.864165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f487ea8-8634-47a0-afc1-22b60a19e52a-ovn-node-metrics-cert\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.865944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.865894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89tx\" (UniqueName: \"kubernetes.io/projected/71460dc1-1655-4123-b9d8-ded03647803a-kube-api-access-g89tx\") pod \"node-ca-mkzlt\" (UID: \"71460dc1-1655-4123-b9d8-ded03647803a\") " pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:57.866992 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.866970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzkm\" (UniqueName: \"kubernetes.io/projected/441030f4-d8a5-48cf-82d4-4882497d5560-kube-api-access-vdzkm\") pod \"node-resolver-jdphr\" (UID: \"441030f4-d8a5-48cf-82d4-4882497d5560\") " pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:57.868080 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.868059 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfkk\" (UniqueName: \"kubernetes.io/projected/2f487ea8-8634-47a0-afc1-22b60a19e52a-kube-api-access-lcfkk\") pod \"ovnkube-node-zsrgc\" (UID: \"2f487ea8-8634-47a0-afc1-22b60a19e52a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:57.868716 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.868698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmnk\" (UniqueName: \"kubernetes.io/projected/eab2cf39-3545-44d8-8eee-3ecefaa5b2e4-kube-api-access-fhmnk\") pod \"tuned-6dpst\" (UID: \"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4\") " pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:57.904574 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.904523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" event={"ID":"cee79eddbd8dbbcb832bb8dad2fba384","Type":"ContainerStarted","Data":"7540234d144ff201b89a9f82f5eae01a91e3d7d5d4655c6c34b8cf107c061d24"} Jul 22 00:26:57.905855 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.905832 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" event={"ID":"f510730f581f273c10f59e1671cc03f4","Type":"ContainerStarted","Data":"824f674f9204951091e5833e0e2b28b1ee7e9c444f645e8d77b14b009571809c"} Jul 22 00:26:57.961766 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.961737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-cnibin\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.961909 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.961779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.961909 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.961850 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-cnibin\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.961909 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.961890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.962056 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.961929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.962056 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.961958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-os-release\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962056 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-cni-bin\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962056 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-cni-multus\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962225 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-cni-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962225 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-socket-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.962225 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-os-release\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962225 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962118 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-cni-bin\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962225 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962137 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-cni-multus\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962225 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-cni-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962264 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-socket-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwgct\" (UniqueName: \"kubernetes.io/projected/8ee4be3b-825a-4893-a581-c0d94530a4cf-kube-api-access-hwgct\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-socket-dir-parent\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962321 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-system-cni-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-netns\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-etc-kubernetes\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-registration-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-hostroot\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-daemon-config\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-kubelet\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b13067b2-8bb6-4faf-8818-e50de72f1db6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-netns\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962426 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-socket-dir-parent\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.962512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-etc-kubernetes\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-system-cni-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-var-lib-kubelet\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962473 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-hostroot\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-kube-api-access-rrcnn\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-registration-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-system-cni-dir\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-sys-fs\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-system-cni-dir\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962683 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-k8s-cni-cncf-io\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.962702 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962706 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-os-release\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-conf-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.962764 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:58.462747464 +0000 UTC m=+3.092979987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-conf-dir\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-device-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-k8s-cni-cncf-io\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-os-release\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr5w\" (UniqueName: \"kubernetes.io/projected/88f41e66-c8b2-485f-809c-6e8f994c4541-kube-api-access-cpr5w\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmbc\" (UniqueName: \"kubernetes.io/projected/21a83aaa-88e2-4c83-9d39-3e393cef3688-kube-api-access-zhmbc\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b13067b2-8bb6-4faf-8818-e50de72f1db6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-host-slash\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962840 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-device-dir\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ee4be3b-825a-4893-a581-c0d94530a4cf-sys-fs\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-host-slash\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21a83aaa-88e2-4c83-9d39-3e393cef3688-cni-binary-copy\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962909 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21a83aaa-88e2-4c83-9d39-3e393cef3688-multus-daemon-config\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.963817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-iptables-alerter-script\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.964513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwmz\" (UniqueName: \"kubernetes.io/projected/b13067b2-8bb6-4faf-8818-e50de72f1db6-kube-api-access-xkwmz\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.964513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.962975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-cnibin\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.964513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.963008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-multus-certs\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.964513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.963063 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-host-run-multus-certs\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.964513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.963066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21a83aaa-88e2-4c83-9d39-3e393cef3688-cnibin\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.964513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.963334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21a83aaa-88e2-4c83-9d39-3e393cef3688-cni-binary-copy\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:57.964513 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.964029 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-iptables-alerter-script\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.968020 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.968002 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:57.968107 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.968024 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:57.968107 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.968036 2562 projected.go:196] Error preparing data for projected volume kube-api-access-d9rgg for pod openshift-network-diagnostics/network-check-target-8r7bq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:57.968217 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:57.968116 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg podName:97d088ad-1080-4d6f-bbb2-92b6eaf278fb nodeName:}" failed. No retries permitted until 2026-07-22 00:26:58.468100656 +0000 UTC m=+3.098333177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d9rgg" (UniqueName: "kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg") pod "network-check-target-8r7bq" (UID: "97d088ad-1080-4d6f-bbb2-92b6eaf278fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:57.970832 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.970810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwgct\" (UniqueName: \"kubernetes.io/projected/8ee4be3b-825a-4893-a581-c0d94530a4cf-kube-api-access-hwgct\") pod \"aws-ebs-csi-driver-node-g94cx\" (UID: \"8ee4be3b-825a-4893-a581-c0d94530a4cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:57.971400 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.971022 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/beae77f6-81b4-4791-a28e-a0d7ebaa2f06-kube-api-access-rrcnn\") pod \"iptables-alerter-27ttx\" (UID: \"beae77f6-81b4-4791-a28e-a0d7ebaa2f06\") " pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:57.971487 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.971460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpr5w\" (UniqueName: \"kubernetes.io/projected/88f41e66-c8b2-485f-809c-6e8f994c4541-kube-api-access-cpr5w\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:57.971651 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.971613 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwmz\" (UniqueName: \"kubernetes.io/projected/b13067b2-8bb6-4faf-8818-e50de72f1db6-kube-api-access-xkwmz\") pod \"multus-additional-cni-plugins-mzrpx\" (UID: \"b13067b2-8bb6-4faf-8818-e50de72f1db6\") " pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:57.972791 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:57.972747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmbc\" (UniqueName: \"kubernetes.io/projected/21a83aaa-88e2-4c83-9d39-3e393cef3688-kube-api-access-zhmbc\") pod \"multus-db4kc\" (UID: \"21a83aaa-88e2-4c83-9d39-3e393cef3688\") " pod="openshift-multus/multus-db4kc" Jul 22 00:26:58.048580 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.048510 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jdphr" Jul 22 00:26:58.055380 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.055225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6dpst" Jul 22 00:26:58.071264 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.071235 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkzlt" Jul 22 00:26:58.075913 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.075894 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:26:58.083154 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.083138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:26:58.089516 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.089495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" Jul 22 00:26:58.100845 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.100827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-27ttx" Jul 22 00:26:58.107278 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.107259 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-db4kc" Jul 22 00:26:58.111803 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.111787 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" Jul 22 00:26:58.120006 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.119988 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:58.365623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.365546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:58.365770 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.365725 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:58.365827 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.365796 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret podName:711d23ce-641f-4bf9-bf7a-cce1d84a05b5 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:59.365778051 +0000 UTC m=+3.996010571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret") pod "global-pull-secret-syncer-dzkpd" (UID: "711d23ce-641f-4bf9-bf7a-cce1d84a05b5") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:58.466364 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.466327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:58.466498 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.466480 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:58.466569 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.466552 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:26:59.466535147 +0000 UTC m=+4.096767675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:58.567395 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.567356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:26:58.567519 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.567504 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:58.567572 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.567525 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:58.567572 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.567534 2562 projected.go:196] Error preparing data for projected volume kube-api-access-d9rgg for pod openshift-network-diagnostics/network-check-target-8r7bq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:58.567649 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:58.567583 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg podName:97d088ad-1080-4d6f-bbb2-92b6eaf278fb nodeName:}" failed. No retries permitted until 2026-07-22 00:26:59.567570535 +0000 UTC m=+4.197803066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9rgg" (UniqueName: "kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg") pod "network-check-target-8r7bq" (UID: "97d088ad-1080-4d6f-bbb2-92b6eaf278fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:58.786154 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.786055 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 00:21:57 +0000 UTC" deadline="2028-01-28 23:20:52.930998095 +0000 UTC" Jul 22 00:26:58.786154 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.786107 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13342h53m54.144893781s" Jul 22 00:26:58.853970 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.853944 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeae77f6_81b4_4791_a28e_a0d7ebaa2f06.slice/crio-dbfbc7b096c058e6934628b3ffb4906ae5c83f5dbf7db2629239ecfc4e55d6f3 WatchSource:0}: Error finding container dbfbc7b096c058e6934628b3ffb4906ae5c83f5dbf7db2629239ecfc4e55d6f3: Status 404 returned error can't find the container with id dbfbc7b096c058e6934628b3ffb4906ae5c83f5dbf7db2629239ecfc4e55d6f3 Jul 22 00:26:58.856064 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.855451 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441030f4_d8a5_48cf_82d4_4882497d5560.slice/crio-42ed0b9b2a01af6ca9b790147e88717587cfe23d74374396693cc90dd5fd0c0a WatchSource:0}: Error finding container 42ed0b9b2a01af6ca9b790147e88717587cfe23d74374396693cc90dd5fd0c0a: Status 404 returned error can't find the container with id 42ed0b9b2a01af6ca9b790147e88717587cfe23d74374396693cc90dd5fd0c0a Jul 22 00:26:58.856577 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.856487 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48b5beb_479e_402e_891a_33a0c4c0449c.slice/crio-4c601a19920a44b429a2ea2058ce4e1c1655b9eed76ce245e2136cfab303e05d WatchSource:0}: Error finding container 4c601a19920a44b429a2ea2058ce4e1c1655b9eed76ce245e2136cfab303e05d: Status 404 returned error can't find the container with id 4c601a19920a44b429a2ea2058ce4e1c1655b9eed76ce245e2136cfab303e05d Jul 22 00:26:58.858970 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.858953 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f487ea8_8634_47a0_afc1_22b60a19e52a.slice/crio-503fe1dbf290d59f3cfe4497a4adbec2ed77014b86eb9784c34f731b91621b1e WatchSource:0}: Error finding container 503fe1dbf290d59f3cfe4497a4adbec2ed77014b86eb9784c34f731b91621b1e: Status 404 returned error can't find the container with id 503fe1dbf290d59f3cfe4497a4adbec2ed77014b86eb9784c34f731b91621b1e Jul 22 00:26:58.859817 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.859802 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee4be3b_825a_4893_a581_c0d94530a4cf.slice/crio-a1643ee767948538a7ce3cd9bc6f067e8edaca7640772b2f2db9d4fcc54cae27 WatchSource:0}: Error finding container a1643ee767948538a7ce3cd9bc6f067e8edaca7640772b2f2db9d4fcc54cae27: Status 404 returned error can't find the container with id a1643ee767948538a7ce3cd9bc6f067e8edaca7640772b2f2db9d4fcc54cae27 Jul 22 00:26:58.879911 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.879890 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab2cf39_3545_44d8_8eee_3ecefaa5b2e4.slice/crio-643c0e8bbce6ae169899963828118e3d3c6940ea9e47bb6ff0094bec5d6a5f3b WatchSource:0}: Error finding container 643c0e8bbce6ae169899963828118e3d3c6940ea9e47bb6ff0094bec5d6a5f3b: Status 404 returned error can't find the container with id 643c0e8bbce6ae169899963828118e3d3c6940ea9e47bb6ff0094bec5d6a5f3b Jul 22 00:26:58.880453 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.880362 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71460dc1_1655_4123_b9d8_ded03647803a.slice/crio-8fdc73f07582646e8065cd71261a68c544389992042efacb3fa895e06ed4df5d WatchSource:0}: Error finding container 8fdc73f07582646e8065cd71261a68c544389992042efacb3fa895e06ed4df5d: Status 404 returned error can't find the container with id 8fdc73f07582646e8065cd71261a68c544389992042efacb3fa895e06ed4df5d Jul 22 00:26:58.883778 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.883683 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13067b2_8bb6_4faf_8818_e50de72f1db6.slice/crio-2ff70fda19922a1ce013edd52b7b6f73196797aa7918445fd6cbd3095af2b963 WatchSource:0}: Error finding container 2ff70fda19922a1ce013edd52b7b6f73196797aa7918445fd6cbd3095af2b963: Status 404 returned error can't find the container with id 2ff70fda19922a1ce013edd52b7b6f73196797aa7918445fd6cbd3095af2b963 Jul 22 00:26:58.884944 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:26:58.884911 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a83aaa_88e2_4c83_9d39_3e393cef3688.slice/crio-da3bfc48b2f865e705d4d31c4408315333c7dc9d441296c975b4a712851dcec4 WatchSource:0}: Error finding container da3bfc48b2f865e705d4d31c4408315333c7dc9d441296c975b4a712851dcec4: Status 404 returned error can't find the container with id da3bfc48b2f865e705d4d31c4408315333c7dc9d441296c975b4a712851dcec4 Jul 22 00:26:58.912535 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.912514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6dpst" event={"ID":"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4","Type":"ContainerStarted","Data":"643c0e8bbce6ae169899963828118e3d3c6940ea9e47bb6ff0094bec5d6a5f3b"} Jul 22 00:26:58.914780 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.914761 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"503fe1dbf290d59f3cfe4497a4adbec2ed77014b86eb9784c34f731b91621b1e"} Jul 22 00:26:58.916196 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.916178 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-db4kc" event={"ID":"21a83aaa-88e2-4c83-9d39-3e393cef3688","Type":"ContainerStarted","Data":"da3bfc48b2f865e705d4d31c4408315333c7dc9d441296c975b4a712851dcec4"} Jul 22 00:26:58.917350 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.917310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerStarted","Data":"2ff70fda19922a1ce013edd52b7b6f73196797aa7918445fd6cbd3095af2b963"} Jul 22 00:26:58.918266 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.918247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkzlt" event={"ID":"71460dc1-1655-4123-b9d8-ded03647803a","Type":"ContainerStarted","Data":"8fdc73f07582646e8065cd71261a68c544389992042efacb3fa895e06ed4df5d"} Jul 22 00:26:58.920492 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.920463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" event={"ID":"8ee4be3b-825a-4893-a581-c0d94530a4cf","Type":"ContainerStarted","Data":"a1643ee767948538a7ce3cd9bc6f067e8edaca7640772b2f2db9d4fcc54cae27"} Jul 22 00:26:58.923624 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.923608 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9ndd" event={"ID":"e48b5beb-479e-402e-891a-33a0c4c0449c","Type":"ContainerStarted","Data":"4c601a19920a44b429a2ea2058ce4e1c1655b9eed76ce245e2136cfab303e05d"} Jul 22 00:26:58.924866 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.924849 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jdphr" event={"ID":"441030f4-d8a5-48cf-82d4-4882497d5560","Type":"ContainerStarted","Data":"42ed0b9b2a01af6ca9b790147e88717587cfe23d74374396693cc90dd5fd0c0a"} Jul 22 00:26:58.925594 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:58.925579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-27ttx" event={"ID":"beae77f6-81b4-4791-a28e-a0d7ebaa2f06","Type":"ContainerStarted","Data":"dbfbc7b096c058e6934628b3ffb4906ae5c83f5dbf7db2629239ecfc4e55d6f3"} Jul 22 00:26:59.373485 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.373155 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:59.373485 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.373379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:59.373485 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.373441 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret podName:711d23ce-641f-4bf9-bf7a-cce1d84a05b5 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:01.373420082 +0000 UTC m=+6.003652605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret") pod "global-pull-secret-syncer-dzkpd" (UID: "711d23ce-641f-4bf9-bf7a-cce1d84a05b5") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:59.474684 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.474652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:59.474815 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.474781 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:59.474866 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.474839 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:01.474823134 +0000 UTC m=+6.105055662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:59.575740 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.575709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:26:59.575896 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.575845 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:59.575896 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.575867 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:59.575896 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.575877 2562 projected.go:196] Error preparing data for projected volume kube-api-access-d9rgg for pod openshift-network-diagnostics/network-check-target-8r7bq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:59.576235 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.576059 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg podName:97d088ad-1080-4d6f-bbb2-92b6eaf278fb nodeName:}" failed. No retries permitted until 2026-07-22 00:27:01.575913319 +0000 UTC m=+6.206145856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9rgg" (UniqueName: "kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg") pod "network-check-target-8r7bq" (UID: "97d088ad-1080-4d6f-bbb2-92b6eaf278fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:59.786660 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.786546 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 00:21:57 +0000 UTC" deadline="2028-01-25 01:31:21.204102763 +0000 UTC" Jul 22 00:26:59.786660 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.786603 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13249h4m21.417503741s" Jul 22 00:26:59.902477 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.901694 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:26:59.902477 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.901814 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:26:59.902477 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.902172 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:26:59.902477 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.902253 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:26:59.902477 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.902309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:26:59.902477 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:26:59.902377 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:26:59.937990 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.937958 2562 generic.go:356] "Generic (PLEG): container finished" podID="f510730f581f273c10f59e1671cc03f4" containerID="d152d62484d93edf78d3874793afd7649131c83e5dee0c7130a57e35b9ae7219" exitCode=0 Jul 22 00:26:59.938679 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.938615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" event={"ID":"f510730f581f273c10f59e1671cc03f4","Type":"ContainerDied","Data":"d152d62484d93edf78d3874793afd7649131c83e5dee0c7130a57e35b9ae7219"} Jul 22 00:26:59.947112 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.946986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" event={"ID":"cee79eddbd8dbbcb832bb8dad2fba384","Type":"ContainerStarted","Data":"9694b96f6eb77825610a9b80e9761dec3f801c8502e6928d69a37f257d449635"} Jul 22 00:26:59.962091 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:26:59.961930 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" podStartSLOduration=3.961914277 podStartE2EDuration="3.961914277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:26:59.960995808 +0000 UTC m=+4.591228345" watchObservedRunningTime="2026-07-22 00:26:59.961914277 +0000 UTC m=+4.592146814" Jul 22 00:27:00.967741 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:00.967706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" event={"ID":"f510730f581f273c10f59e1671cc03f4","Type":"ContainerStarted","Data":"70245301ec2ebff1d0e847d111a58772158b2e3211274976be90acdec2305919"} Jul 22 00:27:00.981138 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:00.981088 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" podStartSLOduration=4.981069728 podStartE2EDuration="4.981069728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:00.980257553 +0000 UTC m=+5.610490094" watchObservedRunningTime="2026-07-22 00:27:00.981069728 +0000 UTC m=+5.611302261" Jul 22 00:27:01.394584 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:01.393975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:01.394584 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.394139 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:01.394584 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.394206 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret podName:711d23ce-641f-4bf9-bf7a-cce1d84a05b5 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:05.394187032 +0000 UTC m=+10.024419553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret") pod "global-pull-secret-syncer-dzkpd" (UID: "711d23ce-641f-4bf9-bf7a-cce1d84a05b5") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:01.495199 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:01.494712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:01.495199 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.494897 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:01.495199 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.494957 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:05.494937251 +0000 UTC m=+10.125169774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:01.595338 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:01.595305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:01.595499 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.595439 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:01.595499 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.595454 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:01.595499 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.595462 2562 projected.go:196] Error preparing data for projected volume kube-api-access-d9rgg for pod openshift-network-diagnostics/network-check-target-8r7bq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:01.595680 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.595500 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg podName:97d088ad-1080-4d6f-bbb2-92b6eaf278fb nodeName:}" failed. No retries permitted until 2026-07-22 00:27:05.595487987 +0000 UTC m=+10.225720509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9rgg" (UniqueName: "kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg") pod "network-check-target-8r7bq" (UID: "97d088ad-1080-4d6f-bbb2-92b6eaf278fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:01.900705 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:01.900624 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:01.900864 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.900765 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:01.900864 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:01.900825 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:01.900965 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:01.900876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:01.901030 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.900978 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:01.901098 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:01.901073 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:03.900833 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:03.900761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:03.901360 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:03.900886 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:03.901360 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:03.901291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:03.901490 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:03.901379 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:03.903102 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:03.902855 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:03.903102 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:03.902973 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:05.426592 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:05.426554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:05.427209 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.426755 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:05.427209 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.426820 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret podName:711d23ce-641f-4bf9-bf7a-cce1d84a05b5 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:13.426802331 +0000 UTC m=+18.057034863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret") pod "global-pull-secret-syncer-dzkpd" (UID: "711d23ce-641f-4bf9-bf7a-cce1d84a05b5") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:05.527695 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:05.527505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:05.527867 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.527712 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:05.527867 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.527779 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:13.527759663 +0000 UTC m=+18.157992185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:05.628927 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:05.628886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:05.629143 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.629095 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:05.629143 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.629122 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:05.629143 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.629137 2562 projected.go:196] Error preparing data for projected volume kube-api-access-d9rgg for pod openshift-network-diagnostics/network-check-target-8r7bq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:05.629346 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.629200 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg podName:97d088ad-1080-4d6f-bbb2-92b6eaf278fb nodeName:}" failed. No retries permitted until 2026-07-22 00:27:13.629178985 +0000 UTC m=+18.259411521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9rgg" (UniqueName: "kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg") pod "network-check-target-8r7bq" (UID: "97d088ad-1080-4d6f-bbb2-92b6eaf278fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:05.900950 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:05.900730 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:05.900950 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.900840 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:05.900950 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:05.900891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:05.901192 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.900949 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:05.901421 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:05.901273 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:05.901421 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:05.901373 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:07.900405 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:07.900366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:07.900867 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:07.900366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:07.900867 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:07.900501 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:07.900867 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:07.900583 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:07.900867 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:07.900367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:07.900867 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:07.900699 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:09.899876 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:09.899837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:09.899876 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:09.899867 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:09.900373 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:09.899837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:09.900373 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:09.899979 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:09.900373 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:09.900089 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:09.900373 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:09.900207 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:11.899726 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:11.899690 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:11.900166 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:11.899736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:11.900166 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:11.899701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:11.900166 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:11.899826 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:11.900166 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:11.899934 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:11.900166 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:11.900027 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:13.486420 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:13.486340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:13.486846 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.486467 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:13.486846 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.486539 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret podName:711d23ce-641f-4bf9-bf7a-cce1d84a05b5 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:29.486519447 +0000 UTC m=+34.116751974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret") pod "global-pull-secret-syncer-dzkpd" (UID: "711d23ce-641f-4bf9-bf7a-cce1d84a05b5") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:13.586979 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:13.586943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:13.587160 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.587141 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:13.587225 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.587213 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:29.587192412 +0000 UTC m=+34.217424936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:13.687942 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:13.687907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:13.688137 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.688065 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:13.688137 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.688084 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:13.688137 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.688096 2562 projected.go:196] Error preparing data for projected volume kube-api-access-d9rgg for pod openshift-network-diagnostics/network-check-target-8r7bq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:13.688254 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.688155 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg podName:97d088ad-1080-4d6f-bbb2-92b6eaf278fb nodeName:}" failed. No retries permitted until 2026-07-22 00:27:29.688136483 +0000 UTC m=+34.318369017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9rgg" (UniqueName: "kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg") pod "network-check-target-8r7bq" (UID: "97d088ad-1080-4d6f-bbb2-92b6eaf278fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:13.899956 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:13.899875 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:13.900160 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:13.899878 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:13.900160 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.899999 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:13.900160 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.900065 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:13.900160 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:13.899878 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:13.900160 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:13.900140 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:15.903163 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:15.903101 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:15.903605 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:15.903208 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:15.903605 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:15.903216 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:15.903605 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:15.903238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:15.903605 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:15.903323 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:15.903605 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:15.903392 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:17.899935 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:17.899756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:17.900396 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:17.899756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:17.900396 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:17.900022 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:17.900396 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:17.900130 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:17.900396 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:17.899753 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:17.900396 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:17.900244 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:18.000440 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.000349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6dpst" event={"ID":"eab2cf39-3545-44d8-8eee-3ecefaa5b2e4","Type":"ContainerStarted","Data":"3af0692be47c773bdce9af8fa0572378793946cb6ee9d1f62d6dcdc5ec5fb558"} Jul 22 00:27:18.003882 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.003859 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:27:18.004317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.004289 2562 generic.go:356] "Generic (PLEG): container finished" podID="2f487ea8-8634-47a0-afc1-22b60a19e52a" containerID="a4f558bb368b5a12c00116bcdbff2210e55f5e894bcdeb816950a0e8a40e7212" exitCode=1 Jul 22 00:27:18.004429 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.004375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"a0fc84309aedc5410cf75cc5ef46bbb9174050aa1985d912a19cc783cb278b48"} Jul 22 00:27:18.004487 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.004452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"bb17c6536e080bc52e1931909f433fc131c2223ee0944c98615b7fdf6b67a8e6"} Jul 22 00:27:18.004536 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.004505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"28e31d77e91a170ec532ea62e480a3f068a86e503654d71d7c8446e4b26a984e"} Jul 22 00:27:18.004582 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.004551 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"0654b941cda1b3f91458da671d5683b5a13a1137c89cb9adc84129735b742cc3"} Jul 22 00:27:18.004662 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.004600 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerDied","Data":"a4f558bb368b5a12c00116bcdbff2210e55f5e894bcdeb816950a0e8a40e7212"} Jul 22 00:27:18.004725 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.004659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"2b4665d3b7f22e15a49cf860c727f12282d9ac79db3dedbed7cf54d08ebc48f2"} Jul 22 00:27:18.005897 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.005865 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-db4kc" event={"ID":"21a83aaa-88e2-4c83-9d39-3e393cef3688","Type":"ContainerStarted","Data":"d4235bb6ec16367b5a96ca15389a07c46e29b92bfca75e8520b5e1256f150463"} Jul 22 00:27:18.007617 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.007590 2562 generic.go:356] "Generic (PLEG): container finished" podID="b13067b2-8bb6-4faf-8818-e50de72f1db6" containerID="d319efa37d080f4811ffc4522dc5b63c50bc12bf1dce74b31b4ff6bfb79d849a" exitCode=0 Jul 22 00:27:18.007734 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.007690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerDied","Data":"d319efa37d080f4811ffc4522dc5b63c50bc12bf1dce74b31b4ff6bfb79d849a"} Jul 22 00:27:18.009578 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.009549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkzlt" event={"ID":"71460dc1-1655-4123-b9d8-ded03647803a","Type":"ContainerStarted","Data":"75a2dd75367164961e01f511f6c891d5c34a2c40e7299408692419d63f64f3c2"} Jul 22 00:27:18.011215 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.011196 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" event={"ID":"8ee4be3b-825a-4893-a581-c0d94530a4cf","Type":"ContainerStarted","Data":"86c17276de27f544e183c7ed9f66526a55b6c8ca037f24b641192ccef5dfe3d6"} Jul 22 00:27:18.012771 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.012737 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9ndd" event={"ID":"e48b5beb-479e-402e-891a-33a0c4c0449c","Type":"ContainerStarted","Data":"e1d1a35f1b6535e7b308af643d5722a97e588c3faab217bce7ca68665d27a2a7"} Jul 22 00:27:18.014416 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.014396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jdphr" event={"ID":"441030f4-d8a5-48cf-82d4-4882497d5560","Type":"ContainerStarted","Data":"2eba05c3c04616f1031a244249ef072dad5b40b518a8c9bda598f8646f77cae5"} Jul 22 00:27:18.017315 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.017269 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6dpst" podStartSLOduration=4.853895702 podStartE2EDuration="23.017258478s" totalImagesPullingTime="18.163362776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.01698315 +0000 UTC m=+22.647215684" watchObservedRunningTime="2026-07-22 00:27:18.017258478 +0000 UTC m=+22.647491019" Jul 22 00:27:18.030281 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.030244 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jdphr" podStartSLOduration=4.843655507 podStartE2EDuration="23.030228934s" totalImagesPullingTime="18.186573427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.030081858 +0000 UTC m=+22.660314589" watchObservedRunningTime="2026-07-22 00:27:18.030228934 +0000 UTC m=+22.660461489" Jul 22 00:27:18.042634 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.042579 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x9ndd" podStartSLOduration=8.907399544 podStartE2EDuration="23.042565292s" totalImagesPullingTime="14.135165748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.042171038 +0000 UTC m=+22.672403577" watchObservedRunningTime="2026-07-22 00:27:18.042565292 +0000 UTC m=+22.672797847" Jul 22 00:27:18.073167 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.073128 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mkzlt" podStartSLOduration=4.943194527 podStartE2EDuration="23.073115341s" totalImagesPullingTime="18.129920814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.072958921 +0000 UTC m=+22.703191461" watchObservedRunningTime="2026-07-22 00:27:18.073115341 +0000 UTC m=+22.703347881" Jul 22 00:27:18.088964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.088925 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-db4kc" podStartSLOduration=4.893640344 podStartE2EDuration="23.088912484s" totalImagesPullingTime="18.19527214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.088813908 +0000 UTC m=+22.719046448" watchObservedRunningTime="2026-07-22 00:27:18.088912484 +0000 UTC m=+22.719145022" Jul 22 00:27:18.249535 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.249512 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:27:18.250264 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.250251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:27:18.493969 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.493942 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 00:27:18.838401 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.838288 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T00:27:18.493965368Z","UUID":"23dd90c9-af7b-4dbc-9b54-0310b9ef8ccd","Handler":null,"Name":"","Endpoint":""} Jul 22 00:27:18.839953 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.839933 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 00:27:18.840070 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:18.839977 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 00:27:19.018846 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.018816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" event={"ID":"8ee4be3b-825a-4893-a581-c0d94530a4cf","Type":"ContainerStarted","Data":"6150220bde48b2a1f2226cfd797a544ea6896da0d39c7c88d9c9f841a0a1b967"} Jul 22 00:27:19.020371 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.020342 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-27ttx" event={"ID":"beae77f6-81b4-4791-a28e-a0d7ebaa2f06","Type":"ContainerStarted","Data":"1fbfc503cd9a29899c729f96cfa55190f37f7a7cb104815656e2514097208343"} Jul 22 00:27:19.021964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.021795 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:27:19.021964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.021883 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x9ndd" Jul 22 00:27:19.035813 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.035760 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-27ttx" podStartSLOduration=5.878592168 podStartE2EDuration="24.035743971s" totalImagesPullingTime="18.157151803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:19.035104435 +0000 UTC m=+23.665336977" watchObservedRunningTime="2026-07-22 00:27:19.035743971 +0000 UTC m=+23.665976512" Jul 22 00:27:19.903163 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.903137 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:19.903304 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.903141 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:19.903304 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:19.903262 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:19.903390 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:19.903140 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:19.903390 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:19.903349 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:19.903465 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:19.903434 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:20.024736 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:20.024702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" event={"ID":"8ee4be3b-825a-4893-a581-c0d94530a4cf","Type":"ContainerStarted","Data":"deafddb5d40777c45ec9acbe618e48f8d034975af096109bda3bf93bf972e870"} Jul 22 00:27:20.048357 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:20.048318 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g94cx" podStartSLOduration=4.522916735 podStartE2EDuration="25.048303769s" totalImagesPullingTime="20.525387034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:20.047983612 +0000 UTC m=+24.678216152" watchObservedRunningTime="2026-07-22 00:27:20.048303769 +0000 UTC m=+24.678536358" Jul 22 00:27:21.030266 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:21.030089 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:27:21.030966 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:21.030665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"2092ed63a1f5a0bcb8d258f1a74160bbcde06ea804822d19d86fd7b1e319d42e"} Jul 22 00:27:21.899573 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:21.899541 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:21.899573 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:21.899579 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:21.899918 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:21.899549 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:21.899918 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:21.899687 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:21.899918 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:21.899745 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:21.899918 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:21.899865 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:23.042205 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.042152 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:27:23.043691 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.042779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"91984962c1bcef587a486b70bc5588fa66a2ec9178c360bd4da7ec1c34f818be"} Jul 22 00:27:23.043691 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.043090 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:27:23.043691 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.043340 2562 scope.go:117] "RemoveContainer" containerID="a4f558bb368b5a12c00116bcdbff2210e55f5e894bcdeb816950a0e8a40e7212" Jul 22 00:27:23.061504 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.061485 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:27:23.902841 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.902561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:23.902964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.902611 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:23.902964 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:23.902852 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:23.902964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:23.902614 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:23.902964 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:23.902912 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:23.903084 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:23.903004 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:24.046813 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.046776 2562 generic.go:356] "Generic (PLEG): container finished" podID="b13067b2-8bb6-4faf-8818-e50de72f1db6" containerID="cc2fb3af47435107dd7cca08bcc4a586226dc29adaa236d127a8258d61e1932c" exitCode=0 Jul 22 00:27:24.047331 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.046873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerDied","Data":"cc2fb3af47435107dd7cca08bcc4a586226dc29adaa236d127a8258d61e1932c"} Jul 22 00:27:24.050679 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.050663 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:27:24.051041 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.051017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" event={"ID":"2f487ea8-8634-47a0-afc1-22b60a19e52a","Type":"ContainerStarted","Data":"8cc5a11a8426900a6f05a3ba353f24bf222300d51668f000936d0cb91e1993ad"} Jul 22 00:27:24.051225 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.051207 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:27:24.051294 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.051281 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:27:24.066119 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.066100 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:27:24.092839 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.092776 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" podStartSLOduration=10.555836748 podStartE2EDuration="29.092759296s" totalImagesPullingTime="18.536922548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:24.092176131 +0000 UTC m=+28.722408670" watchObservedRunningTime="2026-07-22 00:27:24.092759296 +0000 UTC m=+28.722991862" Jul 22 00:27:24.949823 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.949798 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzkpd"] Jul 22 00:27:24.949992 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.949963 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:24.950112 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:24.950082 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:24.950687 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.950664 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9f5n"] Jul 22 00:27:24.950794 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.950776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:24.950906 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:24.950883 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:24.951328 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.951307 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8r7bq"] Jul 22 00:27:24.951417 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:24.951398 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:24.951509 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:24.951489 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:25.055197 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:25.055168 2562 generic.go:356] "Generic (PLEG): container finished" podID="b13067b2-8bb6-4faf-8818-e50de72f1db6" containerID="977815403e9e084a8df7c3cc42858a9c189bd7f4eb296031fb7571326e18ddd0" exitCode=0 Jul 22 00:27:25.055584 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:25.055255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerDied","Data":"977815403e9e084a8df7c3cc42858a9c189bd7f4eb296031fb7571326e18ddd0"} Jul 22 00:27:26.060077 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:26.060042 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerStarted","Data":"4cceff21cd072ebe89c738a0e4144c974ef5d89870e70cf0d709c276a43facd1"} Jul 22 00:27:26.899863 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:26.899613 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:26.900003 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:26.899671 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:26.900003 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:26.899892 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:26.900003 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:26.899686 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:26.900106 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:26.899949 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:26.900106 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:26.900035 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:27.064333 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:27.064299 2562 generic.go:356] "Generic (PLEG): container finished" podID="b13067b2-8bb6-4faf-8818-e50de72f1db6" containerID="4cceff21cd072ebe89c738a0e4144c974ef5d89870e70cf0d709c276a43facd1" exitCode=0 Jul 22 00:27:27.064687 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:27.064340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerDied","Data":"4cceff21cd072ebe89c738a0e4144c974ef5d89870e70cf0d709c276a43facd1"} Jul 22 00:27:28.900334 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:28.900251 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:28.900334 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:28.900279 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:28.900334 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:28.900264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:28.900853 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:28.900395 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r7bq" podUID="97d088ad-1080-4d6f-bbb2-92b6eaf278fb" Jul 22 00:27:28.900853 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:28.900469 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:27:28.900853 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:28.900594 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzkpd" podUID="711d23ce-641f-4bf9-bf7a-cce1d84a05b5" Jul 22 00:27:29.503437 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:29.503396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:29.503671 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.503507 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:29.503671 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.503568 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret podName:711d23ce-641f-4bf9-bf7a-cce1d84a05b5 nodeName:}" failed. No retries permitted until 2026-07-22 00:28:01.503551891 +0000 UTC m=+66.133784428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret") pod "global-pull-secret-syncer-dzkpd" (UID: "711d23ce-641f-4bf9-bf7a-cce1d84a05b5") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:29.604784 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:29.604744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:29.604955 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.604894 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:29.605019 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.604964 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:28:01.604944426 +0000 UTC m=+66.235176962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:29.705760 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:29.705718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:29.705898 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.705832 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:29.705898 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.705858 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:29.705898 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.705868 2562 projected.go:196] Error preparing data for projected volume kube-api-access-d9rgg for pod openshift-network-diagnostics/network-check-target-8r7bq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:29.706044 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:29.705930 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg podName:97d088ad-1080-4d6f-bbb2-92b6eaf278fb nodeName:}" failed. No retries permitted until 2026-07-22 00:28:01.705913599 +0000 UTC m=+66.336146121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d9rgg" (UniqueName: "kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg") pod "network-check-target-8r7bq" (UID: "97d088ad-1080-4d6f-bbb2-92b6eaf278fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:30.136160 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.136128 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeReady" Jul 22 00:27:30.136794 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.136279 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 00:27:30.178579 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.178549 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-44b7j"] Jul 22 00:27:30.206661 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.206619 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zfwgj"] Jul 22 00:27:30.206807 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.206762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.209336 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.209314 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 00:27:30.209469 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.209336 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfkpn\"" Jul 22 00:27:30.209469 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.209456 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 00:27:30.230922 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.230891 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-44b7j"] Jul 22 00:27:30.231042 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.230936 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zfwgj"] Jul 22 00:27:30.231042 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.231024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.233467 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.233446 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 00:27:30.233578 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.233483 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 00:27:30.234521 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.233944 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9frz8\"" Jul 22 00:27:30.234521 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.234015 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 00:27:30.310616 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.310586 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/c995ce90-2e54-4a9e-b74a-e579d7643edc-kube-api-access-9c26w\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.310744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.310635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c995ce90-2e54-4a9e-b74a-e579d7643edc-config-volume\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.310744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.310673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.310744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.310694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cba53b4-96f3-4b32-8f07-9f287fbae77e-cert\") pod \"ingress-canary-zfwgj\" (UID: \"0cba53b4-96f3-4b32-8f07-9f287fbae77e\") " pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.310744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.310723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c995ce90-2e54-4a9e-b74a-e579d7643edc-tmp-dir\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.310744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.310744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm994\" (UniqueName: \"kubernetes.io/projected/0cba53b4-96f3-4b32-8f07-9f287fbae77e-kube-api-access-hm994\") pod \"ingress-canary-zfwgj\" (UID: \"0cba53b4-96f3-4b32-8f07-9f287fbae77e\") " pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.411964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.411891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm994\" (UniqueName: \"kubernetes.io/projected/0cba53b4-96f3-4b32-8f07-9f287fbae77e-kube-api-access-hm994\") pod \"ingress-canary-zfwgj\" (UID: \"0cba53b4-96f3-4b32-8f07-9f287fbae77e\") " pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.411964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.411949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/c995ce90-2e54-4a9e-b74a-e579d7643edc-kube-api-access-9c26w\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.412145 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.412002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c995ce90-2e54-4a9e-b74a-e579d7643edc-config-volume\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.412145 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.412034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.412145 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.412061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cba53b4-96f3-4b32-8f07-9f287fbae77e-cert\") pod \"ingress-canary-zfwgj\" (UID: \"0cba53b4-96f3-4b32-8f07-9f287fbae77e\") " pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.412145 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.412096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c995ce90-2e54-4a9e-b74a-e579d7643edc-tmp-dir\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.412145 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:30.412136 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:30.412298 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:30.412216 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:27:30.912196674 +0000 UTC m=+35.542429205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:27:30.412484 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.412465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c995ce90-2e54-4a9e-b74a-e579d7643edc-tmp-dir\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.412814 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.412791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c995ce90-2e54-4a9e-b74a-e579d7643edc-config-volume\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.416584 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.416562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cba53b4-96f3-4b32-8f07-9f287fbae77e-cert\") pod \"ingress-canary-zfwgj\" (UID: \"0cba53b4-96f3-4b32-8f07-9f287fbae77e\") " pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.420224 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.420148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/c995ce90-2e54-4a9e-b74a-e579d7643edc-kube-api-access-9c26w\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.420552 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.420530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm994\" (UniqueName: \"kubernetes.io/projected/0cba53b4-96f3-4b32-8f07-9f287fbae77e-kube-api-access-hm994\") pod \"ingress-canary-zfwgj\" (UID: \"0cba53b4-96f3-4b32-8f07-9f287fbae77e\") " pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.540708 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.540668 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zfwgj" Jul 22 00:27:30.900185 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.900151 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:27:30.900185 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.900169 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:27:30.900418 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.900169 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:27:30.902761 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.902739 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 00:27:30.902881 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.902778 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 00:27:30.903039 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.903024 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rsxjh\"" Jul 22 00:27:30.903731 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.903709 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 00:27:30.903842 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.903711 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 00:27:30.903842 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.903777 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dh9z6\"" Jul 22 00:27:30.915461 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:30.915442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:30.915614 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:30.915598 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:30.915717 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:30.915705 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:27:31.915688555 +0000 UTC m=+36.545921075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:27:31.922981 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:31.922944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:31.923572 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:31.923107 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:31.923572 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:31.923183 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:27:33.923161849 +0000 UTC m=+38.553394383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:27:32.011719 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:32.011694 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zfwgj"] Jul 22 00:27:32.063602 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:27:32.063574 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cba53b4_96f3_4b32_8f07_9f287fbae77e.slice/crio-e2ec9272f80e0902d9ea5247a5ff6b0d38d42c686774cf99393a408bd4428dc2 WatchSource:0}: Error finding container e2ec9272f80e0902d9ea5247a5ff6b0d38d42c686774cf99393a408bd4428dc2: Status 404 returned error can't find the container with id e2ec9272f80e0902d9ea5247a5ff6b0d38d42c686774cf99393a408bd4428dc2 Jul 22 00:27:32.076100 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:32.075944 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zfwgj" event={"ID":"0cba53b4-96f3-4b32-8f07-9f287fbae77e","Type":"ContainerStarted","Data":"e2ec9272f80e0902d9ea5247a5ff6b0d38d42c686774cf99393a408bd4428dc2"} Jul 22 00:27:33.081840 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:33.081807 2562 generic.go:356] "Generic (PLEG): container finished" podID="b13067b2-8bb6-4faf-8818-e50de72f1db6" containerID="530d09396657737ad86c672b5d55704c347b65be9ed6f9d15c7a274cc43d7390" exitCode=0 Jul 22 00:27:33.082281 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:33.081860 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerDied","Data":"530d09396657737ad86c672b5d55704c347b65be9ed6f9d15c7a274cc43d7390"} Jul 22 00:27:33.937607 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:33.937569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:33.937806 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:33.937696 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:33.937806 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:33.937765 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:27:37.937750166 +0000 UTC m=+42.567982685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:27:34.088795 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:34.088752 2562 generic.go:356] "Generic (PLEG): container finished" podID="b13067b2-8bb6-4faf-8818-e50de72f1db6" containerID="dc806ce949912b4417fc762cfa3f94d6dbe2ea1da929971c4a70a4820aad5ba8" exitCode=0 Jul 22 00:27:34.089170 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:34.088808 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerDied","Data":"dc806ce949912b4417fc762cfa3f94d6dbe2ea1da929971c4a70a4820aad5ba8"} Jul 22 00:27:35.092468 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:35.092251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zfwgj" event={"ID":"0cba53b4-96f3-4b32-8f07-9f287fbae77e","Type":"ContainerStarted","Data":"e71632ab48aa6685d84007611d379a5ab019c29c65d11ecf6942e0a3e2d7cf90"} Jul 22 00:27:35.095233 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:35.095211 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" event={"ID":"b13067b2-8bb6-4faf-8818-e50de72f1db6","Type":"ContainerStarted","Data":"fbcf5e7db08a0da9c37ac55c115d826e7b1c36cbc65245f456f7a1e91075e56f"} Jul 22 00:27:35.106759 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:35.106719 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zfwgj" podStartSLOduration=3.028294132 podStartE2EDuration="5.106700234s" totalImagesPullingTime="2.078406102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:35.106074589 +0000 UTC m=+39.736307130" watchObservedRunningTime="2026-07-22 00:27:35.106700234 +0000 UTC m=+39.736932768" Jul 22 00:27:35.125900 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:35.125852 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mzrpx" podStartSLOduration=10.787386557 podStartE2EDuration="40.125835524s" totalImagesPullingTime="29.338448967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:35.124454606 +0000 UTC m=+39.754687145" watchObservedRunningTime="2026-07-22 00:27:35.125835524 +0000 UTC m=+39.756068065" Jul 22 00:27:37.966743 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:37.966707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:37.967088 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:37.966847 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:37.967088 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:37.966915 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:27:45.966899015 +0000 UTC m=+50.597131534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:27:46.019792 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:46.019754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:27:46.020234 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:46.019882 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:46.020234 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:27:46.019945 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:28:02.019929076 +0000 UTC m=+66.650161595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:27:56.072279 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.072253 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zsrgc" Jul 22 00:27:56.575736 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.575701 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9"] Jul 22 00:27:56.600691 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.600667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9"] Jul 22 00:27:56.600816 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.600763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.602881 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.602853 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 00:27:56.603006 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.602880 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 00:27:56.603006 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.602913 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 00:27:56.603006 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.602959 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 00:27:56.603262 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.603245 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 00:27:56.603823 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.603808 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 00:27:56.603908 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.603860 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 00:27:56.796554 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.796514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.796554 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.796549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.796784 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.796584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d201d3-744d-4361-8589-f8b3429d5923-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.796784 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.796609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-ca\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.796784 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.796671 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkgg\" (UniqueName: \"kubernetes.io/projected/68d201d3-744d-4361-8589-f8b3429d5923-kube-api-access-kmkgg\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.796784 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.796709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-hub\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.897292 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.897214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.897292 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.897247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.897292 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.897280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d201d3-744d-4361-8589-f8b3429d5923-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.897535 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.897400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-ca\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.897535 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.897431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkgg\" (UniqueName: \"kubernetes.io/projected/68d201d3-744d-4361-8589-f8b3429d5923-kube-api-access-kmkgg\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.897535 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.897475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-hub\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.897898 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.897866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d201d3-744d-4361-8589-f8b3429d5923-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.901152 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.901130 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-ca\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.901324 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.901306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-hub\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.901394 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.901302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.901455 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.901438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68d201d3-744d-4361-8589-f8b3429d5923-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.904912 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.904883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkgg\" (UniqueName: \"kubernetes.io/projected/68d201d3-744d-4361-8589-f8b3429d5923-kube-api-access-kmkgg\") pod \"cluster-proxy-proxy-agent-59686576d5-vkwb9\" (UID: \"68d201d3-744d-4361-8589-f8b3429d5923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:56.923866 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:56.923847 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:27:57.054093 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:57.054061 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9"] Jul 22 00:27:57.056737 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:27:57.056701 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68d201d3_744d_4361_8589_f8b3429d5923.slice/crio-eeaeed9b84ad33f69aca7144529accdfc0d8506df3f2cb54bdf13fb39a7a7888 WatchSource:0}: Error finding container eeaeed9b84ad33f69aca7144529accdfc0d8506df3f2cb54bdf13fb39a7a7888: Status 404 returned error can't find the container with id eeaeed9b84ad33f69aca7144529accdfc0d8506df3f2cb54bdf13fb39a7a7888 Jul 22 00:27:57.144625 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:27:57.144597 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" event={"ID":"68d201d3-744d-4361-8589-f8b3429d5923","Type":"ContainerStarted","Data":"eeaeed9b84ad33f69aca7144529accdfc0d8506df3f2cb54bdf13fb39a7a7888"} Jul 22 00:28:01.529823 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.529786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:28:01.532523 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.532501 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 00:28:01.542675 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.542653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/711d23ce-641f-4bf9-bf7a-cce1d84a05b5-original-pull-secret\") pod \"global-pull-secret-syncer-dzkpd\" (UID: \"711d23ce-641f-4bf9-bf7a-cce1d84a05b5\") " pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:28:01.630738 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.630699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:28:01.633185 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.633165 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 00:28:01.641140 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:28:01.641121 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:28:01.641232 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:28:01.641191 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:29:05.641170141 +0000 UTC m=+130.271402659 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : secret "metrics-daemon-secret" not found Jul 22 00:28:01.731757 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.731729 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:28:01.734192 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.734168 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 00:28:01.743799 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.743776 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 00:28:01.754668 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.754634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rgg\" (UniqueName: \"kubernetes.io/projected/97d088ad-1080-4d6f-bbb2-92b6eaf278fb-kube-api-access-d9rgg\") pod \"network-check-target-8r7bq\" (UID: \"97d088ad-1080-4d6f-bbb2-92b6eaf278fb\") " pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:28:01.813654 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.813629 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rsxjh\"" Jul 22 00:28:01.818661 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.818620 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzkpd" Jul 22 00:28:01.821994 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:01.821978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:28:02.034051 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:02.034017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:28:02.034243 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:28:02.034222 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:28:02.034335 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:28:02.034318 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:28:34.034296492 +0000 UTC m=+98.664529025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:28:02.932611 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:02.932449 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzkpd"] Jul 22 00:28:02.935774 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:28:02.935748 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711d23ce_641f_4bf9_bf7a_cce1d84a05b5.slice/crio-80c34ce8b4f828adcee009809509f4e7a338bfae0d5c612d49c1aca1d386e259 WatchSource:0}: Error finding container 80c34ce8b4f828adcee009809509f4e7a338bfae0d5c612d49c1aca1d386e259: Status 404 returned error can't find the container with id 80c34ce8b4f828adcee009809509f4e7a338bfae0d5c612d49c1aca1d386e259 Jul 22 00:28:02.953099 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:02.953078 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8r7bq"] Jul 22 00:28:02.956601 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:28:02.956564 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d088ad_1080_4d6f_bbb2_92b6eaf278fb.slice/crio-1397d0f45639c672dd38a6c9a2314359a4ce4175511919c16519e0a24528da07 WatchSource:0}: Error finding container 1397d0f45639c672dd38a6c9a2314359a4ce4175511919c16519e0a24528da07: Status 404 returned error can't find the container with id 1397d0f45639c672dd38a6c9a2314359a4ce4175511919c16519e0a24528da07 Jul 22 00:28:03.160077 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:03.160042 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8r7bq" event={"ID":"97d088ad-1080-4d6f-bbb2-92b6eaf278fb","Type":"ContainerStarted","Data":"1397d0f45639c672dd38a6c9a2314359a4ce4175511919c16519e0a24528da07"} Jul 22 00:28:03.161012 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:03.160991 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzkpd" event={"ID":"711d23ce-641f-4bf9-bf7a-cce1d84a05b5","Type":"ContainerStarted","Data":"80c34ce8b4f828adcee009809509f4e7a338bfae0d5c612d49c1aca1d386e259"} Jul 22 00:28:03.162721 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:03.162701 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" event={"ID":"68d201d3-744d-4361-8589-f8b3429d5923","Type":"ContainerStarted","Data":"2fba4562d8285658b4767b3de92cd05c854b87df07bfaca87317bdae2228b913"} Jul 22 00:28:03.162798 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:03.162729 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" event={"ID":"68d201d3-744d-4361-8589-f8b3429d5923","Type":"ContainerStarted","Data":"1732180e1734fa0eadab07d30c5c2d2dcdaebe1384d99d422c5c4fd9b9e4e7fb"} Jul 22 00:28:03.162798 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:03.162742 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" event={"ID":"68d201d3-744d-4361-8589-f8b3429d5923","Type":"ContainerStarted","Data":"d705d464bb9f62934e44719ce81deb9202a249a781483f4972aae4afd8b0d59c"} Jul 22 00:28:03.178493 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:03.178450 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" podStartSLOduration=1.398968905 podStartE2EDuration="7.17843232s" totalImagesPullingTime="5.779463415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:28:03.17762815 +0000 UTC m=+67.807860689" watchObservedRunningTime="2026-07-22 00:28:03.17843232 +0000 UTC m=+67.808664857" Jul 22 00:28:08.179743 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:08.179703 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzkpd" event={"ID":"711d23ce-641f-4bf9-bf7a-cce1d84a05b5","Type":"ContainerStarted","Data":"1c801b2ae048257023684871479ede8a6cc5d7ab2d32bb9f88fd5b346564908f"} Jul 22 00:28:08.180985 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:08.180956 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8r7bq" event={"ID":"97d088ad-1080-4d6f-bbb2-92b6eaf278fb","Type":"ContainerStarted","Data":"1dae6fc9b55fedb2c5accd40e985b43feb36cf2c3d77dc217ee50ce452202b71"} Jul 22 00:28:08.181096 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:08.181071 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:28:08.193416 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:08.193370 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dzkpd" podStartSLOduration=68.90362145 podStartE2EDuration="1m13.193358957s" totalImagesPullingTime="4.289737507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:28:08.192732085 +0000 UTC m=+72.822964625" watchObservedRunningTime="2026-07-22 00:28:08.193358957 +0000 UTC m=+72.823591495" Jul 22 00:28:08.205711 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:08.205670 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8r7bq" podStartSLOduration=68.938573147 podStartE2EDuration="1m13.205659511s" totalImagesPullingTime="4.267086364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:28:08.204984953 +0000 UTC m=+72.835217503" watchObservedRunningTime="2026-07-22 00:28:08.205659511 +0000 UTC m=+72.835892042" Jul 22 00:28:34.038240 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:34.038099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:28:34.038728 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:28:34.038264 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:28:34.038728 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:28:34.038348 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:29:38.038326316 +0000 UTC m=+162.668558838 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:28:39.187350 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:28:39.187321 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8r7bq" Jul 22 00:29:05.646838 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:29:05.646798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:29:05.647230 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:29:05.646920 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:29:05.647230 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:29:05.646983 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:07.646968554 +0000 UTC m=+252.277201076 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : secret "metrics-daemon-secret" not found Jul 22 00:29:33.218585 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:29:33.218529 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-44b7j" podUID="c995ce90-2e54-4a9e-b74a-e579d7643edc" Jul 22 00:29:33.413078 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:29:33.413050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44b7j" Jul 22 00:29:33.922720 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:29:33.922677 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:29:38.048511 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:29:38.048479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:29:38.048932 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:29:38.048603 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:29:38.048932 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:29:38.048685 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls podName:c995ce90-2e54-4a9e-b74a-e579d7643edc nodeName:}" failed. No retries permitted until 2026-07-22 00:31:40.048666676 +0000 UTC m=+284.678899198 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls") pod "dns-default-44b7j" (UID: "c995ce90-2e54-4a9e-b74a-e579d7643edc") : secret "dns-default-metrics-tls" not found Jul 22 00:29:48.900368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:29:48.900290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:30:53.848372 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.848338 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw"] Jul 22 00:30:53.850107 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.850092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" Jul 22 00:30:53.852675 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.852655 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 00:30:53.853330 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.853316 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q5nsp\"" Jul 22 00:30:53.853402 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.853380 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 00:30:53.858411 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.858393 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw"] Jul 22 00:30:53.951106 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.951067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ph4\" (UniqueName: \"kubernetes.io/projected/d43ba888-ca31-462d-ba70-5ab908c4990f-kube-api-access-s4ph4\") pod \"migrator-7cc9c7ffd7-sllrw\" (UID: \"d43ba888-ca31-462d-ba70-5ab908c4990f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" Jul 22 00:30:53.986393 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.986362 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rpmxl"] Jul 22 00:30:53.988346 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.988332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:53.990459 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.990439 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 00:30:53.990622 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.990602 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 00:30:53.990732 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.990707 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 00:30:53.990952 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.990935 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qsbbh\"" Jul 22 00:30:53.991029 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.991019 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 00:30:53.999831 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:53.999813 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rpmxl"] Jul 22 00:30:54.052145 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.052121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af504347-b13b-40bd-89ed-3625443321ae-crio-socket\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.052290 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.052157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.052290 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.052186 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fjc\" (UniqueName: \"kubernetes.io/projected/af504347-b13b-40bd-89ed-3625443321ae-kube-api-access-w9fjc\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.052290 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.052252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af504347-b13b-40bd-89ed-3625443321ae-data-volume\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.052414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.052314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.052414 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.052359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ph4\" (UniqueName: \"kubernetes.io/projected/d43ba888-ca31-462d-ba70-5ab908c4990f-kube-api-access-s4ph4\") pod \"migrator-7cc9c7ffd7-sllrw\" (UID: \"d43ba888-ca31-462d-ba70-5ab908c4990f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" Jul 22 00:30:54.059104 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.059080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ph4\" (UniqueName: \"kubernetes.io/projected/d43ba888-ca31-462d-ba70-5ab908c4990f-kube-api-access-s4ph4\") pod \"migrator-7cc9c7ffd7-sllrw\" (UID: \"d43ba888-ca31-462d-ba70-5ab908c4990f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" Jul 22 00:30:54.153549 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.153472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.153549 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.153516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af504347-b13b-40bd-89ed-3625443321ae-crio-socket\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.153549 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.153538 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.153778 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.153561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fjc\" (UniqueName: \"kubernetes.io/projected/af504347-b13b-40bd-89ed-3625443321ae-kube-api-access-w9fjc\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.153778 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.153602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af504347-b13b-40bd-89ed-3625443321ae-data-volume\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.153778 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.153658 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:54.153778 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.153705 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af504347-b13b-40bd-89ed-3625443321ae-crio-socket\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.153778 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.153659 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.153778 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.153728 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:30:54.653710094 +0000 UTC m=+239.283942625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : configmap "kube-rbac-proxy" not found Jul 22 00:30:54.153963 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.153802 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:30:54.65378306 +0000 UTC m=+239.284015593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.153963 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.153898 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af504347-b13b-40bd-89ed-3625443321ae-data-volume\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.157989 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.157974 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" Jul 22 00:30:54.161481 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.161458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fjc\" (UniqueName: \"kubernetes.io/projected/af504347-b13b-40bd-89ed-3625443321ae-kube-api-access-w9fjc\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.276728 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.276701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw"] Jul 22 00:30:54.280255 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:30:54.280230 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43ba888_ca31_462d_ba70_5ab908c4990f.slice/crio-79c1d3e4f8549e1e638024ac426fb2257e0ade7daaae68ac18199d699c42d9a6 WatchSource:0}: Error finding container 79c1d3e4f8549e1e638024ac426fb2257e0ade7daaae68ac18199d699c42d9a6: Status 404 returned error can't find the container with id 79c1d3e4f8549e1e638024ac426fb2257e0ade7daaae68ac18199d699c42d9a6 Jul 22 00:30:54.628618 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.628585 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" event={"ID":"d43ba888-ca31-462d-ba70-5ab908c4990f","Type":"ContainerStarted","Data":"79c1d3e4f8549e1e638024ac426fb2257e0ade7daaae68ac18199d699c42d9a6"} Jul 22 00:30:54.657017 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.656993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.657119 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:54.657047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:54.657154 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.657120 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:54.657154 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.657145 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.657225 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.657182 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:30:55.657164432 +0000 UTC m=+240.287396950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : configmap "kube-rbac-proxy" not found Jul 22 00:30:54.657225 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:54.657202 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:30:55.657195652 +0000 UTC m=+240.287428170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:55.664211 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:55.664171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:55.664592 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:55.664286 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:55.664592 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:55.664317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:55.664592 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:55.664338 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:30:57.664323984 +0000 UTC m=+242.294556502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : configmap "kube-rbac-proxy" not found Jul 22 00:30:55.664592 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:55.664395 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:55.664592 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:55.664437 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:30:57.664425443 +0000 UTC m=+242.294657962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:56.584759 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:56.584734 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jdphr_441030f4-d8a5-48cf-82d4-4882497d5560/dns-node-resolver/0.log" Jul 22 00:30:56.635675 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:56.635627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" event={"ID":"d43ba888-ca31-462d-ba70-5ab908c4990f","Type":"ContainerStarted","Data":"deb9ba44c5fa0304de4f52fb9a74d7554951d6dddcb79ee5fab0895263fb2853"} Jul 22 00:30:56.635811 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:56.635692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" event={"ID":"d43ba888-ca31-462d-ba70-5ab908c4990f","Type":"ContainerStarted","Data":"4e682fb392c36dfe6f76b63615d38a3b2d62b0279886b8c06f8884dbcd88e2d5"} Jul 22 00:30:56.649842 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:56.649798 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-sllrw" podStartSLOduration=1.78237811 podStartE2EDuration="3.649783328s" totalImagesPullingTime="1.867405218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:30:56.648611801 +0000 UTC m=+241.278844340" watchObservedRunningTime="2026-07-22 00:30:56.649783328 +0000 UTC m=+241.280015891" Jul 22 00:30:57.677801 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:57.677766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:57.678180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:57.677835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:30:57.678180 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:57.677906 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:57.678180 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:57.677934 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:57.678180 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:57.677970 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:31:01.677951768 +0000 UTC m=+246.308184287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : configmap "kube-rbac-proxy" not found Jul 22 00:30:57.678180 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:30:57.677986 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:31:01.67797796 +0000 UTC m=+246.308210478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:57.984979 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:57.984912 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkzlt_71460dc1-1655-4123-b9d8-ded03647803a/node-ca/0.log" Jul 22 00:30:58.786436 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:58.786407 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zfwgj_0cba53b4-96f3-4b32-8f07-9f287fbae77e/serve-healthcheck-canary/0.log" Jul 22 00:30:58.985921 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:58.985897 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sllrw_d43ba888-ca31-462d-ba70-5ab908c4990f/migrator/0.log" Jul 22 00:30:59.184774 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:30:59.184745 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sllrw_d43ba888-ca31-462d-ba70-5ab908c4990f/graceful-termination/0.log" Jul 22 00:31:01.705152 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:01.705116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:01.705565 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:01.705204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:01.705565 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:01.705281 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:31:01.705565 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:01.705314 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:31:01.705565 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:01.705340 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:31:09.705324605 +0000 UTC m=+254.335557123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : secret "insights-runtime-extractor-tls" not found Jul 22 00:31:01.705565 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:01.705358 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:31:09.705346891 +0000 UTC m=+254.335579413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : configmap "kube-rbac-proxy" not found Jul 22 00:31:07.652157 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:07.652096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:31:07.652557 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:07.652246 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:31:07.652557 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:07.652303 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs podName:88f41e66-c8b2-485f-809c-6e8f994c4541 nodeName:}" failed. No retries permitted until 2026-07-22 00:33:09.652290547 +0000 UTC m=+374.282523065 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs") pod "network-metrics-daemon-x9f5n" (UID: "88f41e66-c8b2-485f-809c-6e8f994c4541") : secret "metrics-daemon-secret" not found Jul 22 00:31:09.768174 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:09.768137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:09.768560 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:09.768198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:09.768560 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:09.768277 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:31:09.768560 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:09.768333 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls podName:af504347-b13b-40bd-89ed-3625443321ae nodeName:}" failed. No retries permitted until 2026-07-22 00:31:25.768318258 +0000 UTC m=+270.398550777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpmxl" (UID: "af504347-b13b-40bd-89ed-3625443321ae") : secret "insights-runtime-extractor-tls" not found Jul 22 00:31:09.768704 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:09.768688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af504347-b13b-40bd-89ed-3625443321ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:23.493065 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.492987 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6"] Jul 22 00:31:23.494725 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.494711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" Jul 22 00:31:23.496564 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.496540 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 22 00:31:23.496806 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.496790 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8m4qr\"" Jul 22 00:31:23.502404 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.502380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6"] Jul 22 00:31:23.570440 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.570401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bfbbacb0-1be5-40ea-be1d-829bb300f392-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-4m9j6\" (UID: \"bfbbacb0-1be5-40ea-be1d-829bb300f392\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" Jul 22 00:31:23.671660 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.671602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bfbbacb0-1be5-40ea-be1d-829bb300f392-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-4m9j6\" (UID: \"bfbbacb0-1be5-40ea-be1d-829bb300f392\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" Jul 22 00:31:23.674133 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.674104 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bfbbacb0-1be5-40ea-be1d-829bb300f392-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-4m9j6\" (UID: \"bfbbacb0-1be5-40ea-be1d-829bb300f392\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" Jul 22 00:31:23.803753 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.803724 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" Jul 22 00:31:23.926220 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:23.926176 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6"] Jul 22 00:31:23.929823 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:23.929793 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbbacb0_1be5_40ea_be1d_829bb300f392.slice/crio-e26808e0bfd7be27b8c142a9baf0fc0d007158307806ea913d32c564be813166 WatchSource:0}: Error finding container e26808e0bfd7be27b8c142a9baf0fc0d007158307806ea913d32c564be813166: Status 404 returned error can't find the container with id e26808e0bfd7be27b8c142a9baf0fc0d007158307806ea913d32c564be813166 Jul 22 00:31:24.715038 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:24.714999 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" event={"ID":"bfbbacb0-1be5-40ea-be1d-829bb300f392","Type":"ContainerStarted","Data":"e26808e0bfd7be27b8c142a9baf0fc0d007158307806ea913d32c564be813166"} Jul 22 00:31:25.719806 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.719772 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" event={"ID":"bfbbacb0-1be5-40ea-be1d-829bb300f392","Type":"ContainerStarted","Data":"8cb8b2404989439b70d4249e06b88bf88f24041752edc8742cd4d896aca7b151"} Jul 22 00:31:25.720259 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.719908 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" Jul 22 00:31:25.724295 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.724268 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" Jul 22 00:31:25.750503 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.750458 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4m9j6" podStartSLOduration=1.538802171 podStartE2EDuration="2.750443307s" totalImagesPullingTime="1.211641136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:25.735224364 +0000 UTC m=+270.365456905" watchObservedRunningTime="2026-07-22 00:31:25.750443307 +0000 UTC m=+270.380675847" Jul 22 00:31:25.789243 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.789210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:25.791475 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.791442 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af504347-b13b-40bd-89ed-3625443321ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpmxl\" (UID: \"af504347-b13b-40bd-89ed-3625443321ae\") " pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:25.799348 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.799324 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qsbbh\"" Jul 22 00:31:25.807385 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.807363 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rpmxl" Jul 22 00:31:25.929101 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:25.929070 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rpmxl"] Jul 22 00:31:25.932005 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:25.931979 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf504347_b13b_40bd_89ed_3625443321ae.slice/crio-7b680da8b34da2d61108f7fdd3757052597ea4f153c4cd0054a540788fab76af WatchSource:0}: Error finding container 7b680da8b34da2d61108f7fdd3757052597ea4f153c4cd0054a540788fab76af: Status 404 returned error can't find the container with id 7b680da8b34da2d61108f7fdd3757052597ea4f153c4cd0054a540788fab76af Jul 22 00:31:26.725366 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:26.725325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpmxl" event={"ID":"af504347-b13b-40bd-89ed-3625443321ae","Type":"ContainerStarted","Data":"7b680da8b34da2d61108f7fdd3757052597ea4f153c4cd0054a540788fab76af"} Jul 22 00:31:31.746881 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:31.746843 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpmxl" event={"ID":"af504347-b13b-40bd-89ed-3625443321ae","Type":"ContainerStarted","Data":"754a4bb7b6965604f4a700ffbfa236b27126cb1595c3bd60307695ffab7a154d"} Jul 22 00:31:33.886862 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.886830 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-frwrs"] Jul 22 00:31:33.908422 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.908384 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:33.910734 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.910709 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 00:31:33.910870 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.910748 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 00:31:33.910957 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.910941 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 00:31:33.911663 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.911630 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-49khf\"" Jul 22 00:31:33.911663 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.911635 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 00:31:33.912196 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.911994 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 00:31:33.912196 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:33.911997 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 00:31:34.054366 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054556 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-sys\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054556 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm2wt\" (UniqueName: \"kubernetes.io/projected/f308f250-c886-4296-8b2a-e8e820b1dc6c-kube-api-access-nm2wt\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054556 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054730 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-root\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054730 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-wtmp\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054730 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-textfile\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054730 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054699 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f308f250-c886-4296-8b2a-e8e820b1dc6c-metrics-client-ca\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.054730 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.054725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.155874 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.155789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.155874 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.155830 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-root\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.155874 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.155857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-wtmp\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156139 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.155921 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-root\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156139 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.155992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-wtmp\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156139 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-textfile\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156139 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f308f250-c886-4296-8b2a-e8e820b1dc6c-metrics-client-ca\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156139 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-sys\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm2wt\" (UniqueName: \"kubernetes.io/projected/f308f250-c886-4296-8b2a-e8e820b1dc6c-kube-api-access-nm2wt\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156257 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-textfile\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156365 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:34.156286 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 00:31:34.156365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156310 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f308f250-c886-4296-8b2a-e8e820b1dc6c-sys\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156365 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:34.156352 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls podName:f308f250-c886-4296-8b2a-e8e820b1dc6c nodeName:}" failed. No retries permitted until 2026-07-22 00:31:34.656330489 +0000 UTC m=+279.286563024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls") pod "node-exporter-frwrs" (UID: "f308f250-c886-4296-8b2a-e8e820b1dc6c") : secret "node-exporter-tls" not found Jul 22 00:31:34.156676 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156609 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.156724 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.156688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f308f250-c886-4296-8b2a-e8e820b1dc6c-metrics-client-ca\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.158203 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.158182 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.163848 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.163825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm2wt\" (UniqueName: \"kubernetes.io/projected/f308f250-c886-4296-8b2a-e8e820b1dc6c-kube-api-access-nm2wt\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.660199 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:34.660150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:34.660381 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:34.660305 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 00:31:34.660381 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:34.660372 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls podName:f308f250-c886-4296-8b2a-e8e820b1dc6c nodeName:}" failed. No retries permitted until 2026-07-22 00:31:35.66035515 +0000 UTC m=+280.290587671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls") pod "node-exporter-frwrs" (UID: "f308f250-c886-4296-8b2a-e8e820b1dc6c") : secret "node-exporter-tls" not found Jul 22 00:31:35.668134 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.668091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:35.670838 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.670815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f308f250-c886-4296-8b2a-e8e820b1dc6c-node-exporter-tls\") pod \"node-exporter-frwrs\" (UID: \"f308f250-c886-4296-8b2a-e8e820b1dc6c\") " pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:35.719394 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.719362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frwrs" Jul 22 00:31:35.728938 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:35.728913 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf308f250_c886_4296_8b2a_e8e820b1dc6c.slice/crio-05894f36a3ffde9eeb1c1a04ea0cd868eaecef1669609412b3f4a470a5a8233b WatchSource:0}: Error finding container 05894f36a3ffde9eeb1c1a04ea0cd868eaecef1669609412b3f4a470a5a8233b: Status 404 returned error can't find the container with id 05894f36a3ffde9eeb1c1a04ea0cd868eaecef1669609412b3f4a470a5a8233b Jul 22 00:31:35.761095 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.761067 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpmxl" event={"ID":"af504347-b13b-40bd-89ed-3625443321ae","Type":"ContainerStarted","Data":"85cd3d6c8feb4755b1a7d6ced989c5c35cb5c59a3ac4954108b46cd56e30d121"} Jul 22 00:31:35.762112 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.762094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frwrs" event={"ID":"f308f250-c886-4296-8b2a-e8e820b1dc6c","Type":"ContainerStarted","Data":"05894f36a3ffde9eeb1c1a04ea0cd868eaecef1669609412b3f4a470a5a8233b"} Jul 22 00:31:35.875508 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.875474 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67f766cb47-dwg8j"] Jul 22 00:31:35.879966 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.879924 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.882943 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.882391 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-epm829844emfg\"" Jul 22 00:31:35.882943 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.882392 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 00:31:35.882943 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.882614 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vxnxd\"" Jul 22 00:31:35.882943 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.882747 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 00:31:35.882943 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.882797 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 00:31:35.882943 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.882946 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 00:31:35.883267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.883036 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 00:31:35.890229 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.890204 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67f766cb47-dwg8j"] Jul 22 00:31:35.971052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.970971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6vv\" (UniqueName: \"kubernetes.io/projected/411b43ef-a9a2-4221-8057-db8435d41fea-kube-api-access-fd6vv\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.971208 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.971052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.971208 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.971110 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-tls\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.971208 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.971152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.971208 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.971188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-grpc-tls\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.971419 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.971239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.971419 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.971296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:35.971419 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:35.971325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/411b43ef-a9a2-4221-8057-db8435d41fea-metrics-client-ca\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.071730 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-tls\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.071923 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.071923 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071774 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-grpc-tls\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.071923 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071797 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.071923 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.071923 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/411b43ef-a9a2-4221-8057-db8435d41fea-metrics-client-ca\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.071923 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6vv\" (UniqueName: \"kubernetes.io/projected/411b43ef-a9a2-4221-8057-db8435d41fea-kube-api-access-fd6vv\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.072221 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.071961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.073053 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.073000 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/411b43ef-a9a2-4221-8057-db8435d41fea-metrics-client-ca\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.074851 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.074830 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.074971 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.074850 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-grpc-tls\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.074971 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.074957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-tls\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.075205 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.075164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.075457 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.075433 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.075638 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.075619 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/411b43ef-a9a2-4221-8057-db8435d41fea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.080593 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.080572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6vv\" (UniqueName: \"kubernetes.io/projected/411b43ef-a9a2-4221-8057-db8435d41fea-kube-api-access-fd6vv\") pod \"thanos-querier-67f766cb47-dwg8j\" (UID: \"411b43ef-a9a2-4221-8057-db8435d41fea\") " pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.192496 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.192456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:36.330122 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.330092 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67f766cb47-dwg8j"] Jul 22 00:31:36.333077 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:36.333054 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411b43ef_a9a2_4221_8057_db8435d41fea.slice/crio-21ae36cc70ed57e0fc808b4929e91028e5416d62430e04cadf75a9a9caafe81f WatchSource:0}: Error finding container 21ae36cc70ed57e0fc808b4929e91028e5416d62430e04cadf75a9a9caafe81f: Status 404 returned error can't find the container with id 21ae36cc70ed57e0fc808b4929e91028e5416d62430e04cadf75a9a9caafe81f Jul 22 00:31:36.413502 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:36.413466 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-44b7j" podUID="c995ce90-2e54-4a9e-b74a-e579d7643edc" Jul 22 00:31:36.767495 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.767454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" event={"ID":"411b43ef-a9a2-4221-8057-db8435d41fea","Type":"ContainerStarted","Data":"21ae36cc70ed57e0fc808b4929e91028e5416d62430e04cadf75a9a9caafe81f"} Jul 22 00:31:36.767980 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:36.767512 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44b7j" Jul 22 00:31:38.282014 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.281969 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69c588d6d6-kspc5"] Jul 22 00:31:38.309112 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.309078 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69c588d6d6-kspc5"] Jul 22 00:31:38.309305 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.309206 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.312179 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.312155 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cvd1051srdr1p\"" Jul 22 00:31:38.312396 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.312377 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 00:31:38.312932 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.312890 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 00:31:38.313060 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.313029 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-c7cb2\"" Jul 22 00:31:38.313340 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.313322 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 00:31:38.313578 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.313562 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 00:31:38.393688 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.393626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-secret-metrics-server-tls\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.393844 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.393698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-client-ca-bundle\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.393844 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.393722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhvt\" (UniqueName: \"kubernetes.io/projected/5d8a5554-103d-4375-81cb-003322c5d5fb-kube-api-access-8jhvt\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.393844 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.393747 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5d8a5554-103d-4375-81cb-003322c5d5fb-audit-log\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.393844 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.393831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d8a5554-103d-4375-81cb-003322c5d5fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.394029 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.393898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-secret-metrics-server-client-certs\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.394029 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.393922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5d8a5554-103d-4375-81cb-003322c5d5fb-metrics-server-audit-profiles\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.494525 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.494460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-client-ca-bundle\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.494525 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.494529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhvt\" (UniqueName: \"kubernetes.io/projected/5d8a5554-103d-4375-81cb-003322c5d5fb-kube-api-access-8jhvt\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.494787 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.494566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5d8a5554-103d-4375-81cb-003322c5d5fb-audit-log\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.494787 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.494599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d8a5554-103d-4375-81cb-003322c5d5fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.494787 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.494631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-secret-metrics-server-client-certs\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.494787 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.494674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5d8a5554-103d-4375-81cb-003322c5d5fb-metrics-server-audit-profiles\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.494787 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.494760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-secret-metrics-server-tls\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.495039 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.495010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5d8a5554-103d-4375-81cb-003322c5d5fb-audit-log\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.495425 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.495397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d8a5554-103d-4375-81cb-003322c5d5fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.495638 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.495620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5d8a5554-103d-4375-81cb-003322c5d5fb-metrics-server-audit-profiles\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.496941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.496917 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-secret-metrics-server-client-certs\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.497056 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.497039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-secret-metrics-server-tls\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.497117 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.497055 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5d8a5554-103d-4375-81cb-003322c5d5fb-client-ca-bundle\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.502588 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.502567 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhvt\" (UniqueName: \"kubernetes.io/projected/5d8a5554-103d-4375-81cb-003322c5d5fb-kube-api-access-8jhvt\") pod \"metrics-server-69c588d6d6-kspc5\" (UID: \"5d8a5554-103d-4375-81cb-003322c5d5fb\") " pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.619054 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.619028 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:38.668942 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.668905 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh"] Jul 22 00:31:38.673190 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.673166 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:38.675553 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.675333 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 00:31:38.675553 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.675342 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8gntr\"" Jul 22 00:31:38.681614 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.681556 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh"] Jul 22 00:31:38.773029 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.772956 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69c588d6d6-kspc5"] Jul 22 00:31:38.777390 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:38.777335 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8a5554_103d_4375_81cb_003322c5d5fb.slice/crio-42e3f31f467cdc35789833ea35d25027a5fe257e8618d16701ad52f8967c23b1 WatchSource:0}: Error finding container 42e3f31f467cdc35789833ea35d25027a5fe257e8618d16701ad52f8967c23b1: Status 404 returned error can't find the container with id 42e3f31f467cdc35789833ea35d25027a5fe257e8618d16701ad52f8967c23b1 Jul 22 00:31:38.779946 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.779915 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpmxl" event={"ID":"af504347-b13b-40bd-89ed-3625443321ae","Type":"ContainerStarted","Data":"2c2bb812ed5167c9022c296cefe9fd062125a32ec05b05a95d242f7826692c4c"} Jul 22 00:31:38.796000 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.795930 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rpmxl" podStartSLOduration=33.30779701 podStartE2EDuration="45.795863069s" totalImagesPullingTime="12.488066059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:30:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:38.795066956 +0000 UTC m=+283.425299513" watchObservedRunningTime="2026-07-22 00:31:38.795863069 +0000 UTC m=+283.426095610" Jul 22 00:31:38.796935 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.796912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5np\" (UniqueName: \"kubernetes.io/projected/64a31bc3-e4e2-47ef-a515-c1bef899f354-kube-api-access-2w5np\") pod \"monitoring-plugin-7b7b7bc8bf-45wlh\" (UID: \"64a31bc3-e4e2-47ef-a515-c1bef899f354\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:38.797032 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.797008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/64a31bc3-e4e2-47ef-a515-c1bef899f354-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-45wlh\" (UID: \"64a31bc3-e4e2-47ef-a515-c1bef899f354\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:38.898370 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.898333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/64a31bc3-e4e2-47ef-a515-c1bef899f354-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-45wlh\" (UID: \"64a31bc3-e4e2-47ef-a515-c1bef899f354\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:38.898536 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.898424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5np\" (UniqueName: \"kubernetes.io/projected/64a31bc3-e4e2-47ef-a515-c1bef899f354-kube-api-access-2w5np\") pod \"monitoring-plugin-7b7b7bc8bf-45wlh\" (UID: \"64a31bc3-e4e2-47ef-a515-c1bef899f354\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:38.898536 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:38.898497 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 22 00:31:38.898628 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:38.898568 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a31bc3-e4e2-47ef-a515-c1bef899f354-monitoring-plugin-cert podName:64a31bc3-e4e2-47ef-a515-c1bef899f354 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:39.398548496 +0000 UTC m=+284.028781019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/64a31bc3-e4e2-47ef-a515-c1bef899f354-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-45wlh" (UID: "64a31bc3-e4e2-47ef-a515-c1bef899f354") : secret "monitoring-plugin-cert" not found Jul 22 00:31:38.907212 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:38.907185 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5np\" (UniqueName: \"kubernetes.io/projected/64a31bc3-e4e2-47ef-a515-c1bef899f354-kube-api-access-2w5np\") pod \"monitoring-plugin-7b7b7bc8bf-45wlh\" (UID: \"64a31bc3-e4e2-47ef-a515-c1bef899f354\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:39.191404 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.191371 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-857dbfbcbc-74ss7"] Jul 22 00:31:39.195221 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.195199 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.198301 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.197952 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 00:31:39.198301 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.198171 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 00:31:39.198301 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.198207 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 00:31:39.198553 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.198448 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 00:31:39.198610 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.198583 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t9krq\"" Jul 22 00:31:39.198682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.198669 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 00:31:39.198740 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.198704 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 00:31:39.198740 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.198587 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 00:31:39.205062 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.205040 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 00:31:39.206574 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.206531 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857dbfbcbc-74ss7"] Jul 22 00:31:39.302191 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.302153 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-config\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.302623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.302291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-oauth-config\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.302623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.302333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-serving-cert\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.302623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.302360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcfc\" (UniqueName: \"kubernetes.io/projected/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-kube-api-access-8wcfc\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.302623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.302381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-oauth-serving-cert\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.302623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.302396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-trusted-ca-bundle\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.302623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.302484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-service-ca\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.403083 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.402993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-oauth-config\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.403083 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-serving-cert\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.403270 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcfc\" (UniqueName: \"kubernetes.io/projected/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-kube-api-access-8wcfc\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.403270 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-oauth-serving-cert\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.403270 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-trusted-ca-bundle\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.403425 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-service-ca\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.403425 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/64a31bc3-e4e2-47ef-a515-c1bef899f354-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-45wlh\" (UID: \"64a31bc3-e4e2-47ef-a515-c1bef899f354\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:39.403425 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-config\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.404004 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.403977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-oauth-serving-cert\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.404119 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.404032 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-config\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.404493 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.404468 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-service-ca\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.405052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.405026 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-trusted-ca-bundle\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.405828 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.405805 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-oauth-config\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.406031 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.406009 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-serving-cert\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.406164 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.406143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/64a31bc3-e4e2-47ef-a515-c1bef899f354-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-45wlh\" (UID: \"64a31bc3-e4e2-47ef-a515-c1bef899f354\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:39.410460 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.410435 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcfc\" (UniqueName: \"kubernetes.io/projected/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-kube-api-access-8wcfc\") pod \"console-857dbfbcbc-74ss7\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.506836 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.506806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:39.587442 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.587412 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:39.651432 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.650872 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857dbfbcbc-74ss7"] Jul 22 00:31:39.655346 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:39.655278 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c5d22b_52b1_43a3_aba3_ddb4910b3b66.slice/crio-5e709f1f46b93b11a5ed7e2aa9d3a9dd8025350e31d03efdb6f7e6427ab77e21 WatchSource:0}: Error finding container 5e709f1f46b93b11a5ed7e2aa9d3a9dd8025350e31d03efdb6f7e6427ab77e21: Status 404 returned error can't find the container with id 5e709f1f46b93b11a5ed7e2aa9d3a9dd8025350e31d03efdb6f7e6427ab77e21 Jul 22 00:31:39.737064 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.737036 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh"] Jul 22 00:31:39.785408 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.785357 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857dbfbcbc-74ss7" event={"ID":"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66","Type":"ContainerStarted","Data":"5e709f1f46b93b11a5ed7e2aa9d3a9dd8025350e31d03efdb6f7e6427ab77e21"} Jul 22 00:31:39.787258 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.787220 2562 generic.go:356] "Generic (PLEG): container finished" podID="f308f250-c886-4296-8b2a-e8e820b1dc6c" containerID="e92badc7bf77903c3c2754307dc080445f517a66e36d5bba099d8bcd330d2e21" exitCode=0 Jul 22 00:31:39.787398 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.787309 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frwrs" event={"ID":"f308f250-c886-4296-8b2a-e8e820b1dc6c","Type":"ContainerDied","Data":"e92badc7bf77903c3c2754307dc080445f517a66e36d5bba099d8bcd330d2e21"} Jul 22 00:31:39.788783 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:39.788743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" event={"ID":"5d8a5554-103d-4375-81cb-003322c5d5fb","Type":"ContainerStarted","Data":"42e3f31f467cdc35789833ea35d25027a5fe257e8618d16701ad52f8967c23b1"} Jul 22 00:31:40.070431 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.070397 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:31:40.073726 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.073698 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.076030 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.076004 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 00:31:40.076160 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.076128 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 00:31:40.076246 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.076226 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 00:31:40.076593 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.076570 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 00:31:40.076761 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.076737 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 00:31:40.076889 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.076842 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 00:31:40.077062 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077046 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 00:31:40.077062 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077092 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qtt9l\"" Jul 22 00:31:40.077555 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077312 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 00:31:40.077555 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077361 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 00:31:40.077555 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077412 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-db7khhrrkfi27\"" Jul 22 00:31:40.077555 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077459 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 00:31:40.077555 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077478 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 00:31:40.077928 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077712 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 00:31:40.077928 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.077743 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 00:31:40.079163 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.079142 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 00:31:40.090905 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.090821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:31:40.110897 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.110766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:31:40.114146 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.114115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c995ce90-2e54-4a9e-b74a-e579d7643edc-metrics-tls\") pod \"dns-default-44b7j\" (UID: \"c995ce90-2e54-4a9e-b74a-e579d7643edc\") " pod="openshift-dns/dns-default-44b7j" Jul 22 00:31:40.211519 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211487 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211519 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211796 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211796 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211587 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr4v\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-kube-api-access-vsr4v\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211796 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211796 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211796 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211796 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.211796 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211855 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-config\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.211974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.212007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-web-config\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.212027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-config-out\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212121 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.212090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212451 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.212184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212451 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.212205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.212451 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.212229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.312873 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.312831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-web-config\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.312882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-config-out\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.312908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.312952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.312978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr4v\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-kube-api-access-vsr4v\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.313352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.314129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-config\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.314129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313448 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.314129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.314129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.314129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.313950 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.314129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.314036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.314660 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.314506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.315857 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.315325 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.316486 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.316114 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.316486 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.316238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-config-out\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.316486 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.316362 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.316730 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.316708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.316872 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.316851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-web-config\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.317110 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.317087 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.317803 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.317474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.317803 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.317745 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.317950 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.317811 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.318613 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.318590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.318719 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.318599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.318719 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.318698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.319630 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.319610 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-config\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.319902 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.319881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.320088 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.320070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.325157 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.325106 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr4v\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-kube-api-access-vsr4v\") pod \"prometheus-k8s-0\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.371327 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.371295 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfkpn\"" Jul 22 00:31:40.379208 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.379183 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-44b7j" Jul 22 00:31:40.384782 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.384761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:40.591006 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:40.590934 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a31bc3_e4e2_47ef_a515_c1bef899f354.slice/crio-9c4b2e7950d7a42ddaf27a48bfd4a32da4b454c91f3ea78b96076a83a2e5d73b WatchSource:0}: Error finding container 9c4b2e7950d7a42ddaf27a48bfd4a32da4b454c91f3ea78b96076a83a2e5d73b: Status 404 returned error can't find the container with id 9c4b2e7950d7a42ddaf27a48bfd4a32da4b454c91f3ea78b96076a83a2e5d73b Jul 22 00:31:40.802422 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.802329 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" event={"ID":"64a31bc3-e4e2-47ef-a515-c1bef899f354","Type":"ContainerStarted","Data":"9c4b2e7950d7a42ddaf27a48bfd4a32da4b454c91f3ea78b96076a83a2e5d73b"} Jul 22 00:31:40.806717 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.806614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frwrs" event={"ID":"f308f250-c886-4296-8b2a-e8e820b1dc6c","Type":"ContainerStarted","Data":"0c8160f80a481a865be8e27e010101b0491c70d3265b5b9b81dfdfb2a023d0c5"} Jul 22 00:31:40.810570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.810088 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" event={"ID":"411b43ef-a9a2-4221-8057-db8435d41fea","Type":"ContainerStarted","Data":"b4734885ca7d5c64f3fe4a751c6ed79e732573ac80a966bab6ee75d09e28fc9b"} Jul 22 00:31:40.811688 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.811615 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-44b7j"] Jul 22 00:31:40.812347 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.811948 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" event={"ID":"5d8a5554-103d-4375-81cb-003322c5d5fb","Type":"ContainerStarted","Data":"4bd6fd5a9a524af8af5bed1325e6c3b1165803e7c789c5371229eba71c4d6af2"} Jul 22 00:31:40.817943 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:31:40.817914 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc995ce90_2e54_4a9e_b74a_e579d7643edc.slice/crio-71f85dfd1d7f178cb9d4292ad84b61bfb847503352e48ff48c2e428046fb8d22 WatchSource:0}: Error finding container 71f85dfd1d7f178cb9d4292ad84b61bfb847503352e48ff48c2e428046fb8d22: Status 404 returned error can't find the container with id 71f85dfd1d7f178cb9d4292ad84b61bfb847503352e48ff48c2e428046fb8d22 Jul 22 00:31:40.830512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.830308 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" podStartSLOduration=0.96184353 podStartE2EDuration="2.830289344s" totalImagesPullingTime="1.868445814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:40.829481098 +0000 UTC m=+285.459713630" watchObservedRunningTime="2026-07-22 00:31:40.830289344 +0000 UTC m=+285.460521882" Jul 22 00:31:40.854522 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:40.854345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:31:41.818964 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:41.818926 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerStarted","Data":"d1e6b94b29c0d68c6a068dfe7d403c37832a5733e458ea1f8c55427c1c5a4937"} Jul 22 00:31:41.822791 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:41.822759 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frwrs" event={"ID":"f308f250-c886-4296-8b2a-e8e820b1dc6c","Type":"ContainerStarted","Data":"3e7e4e0cd6256a471d4e1bcc6b4fe2860acb3426862dc549217f99f4f541de2d"} Jul 22 00:31:41.826579 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:41.826553 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" event={"ID":"411b43ef-a9a2-4221-8057-db8435d41fea","Type":"ContainerStarted","Data":"929f1b0c7ee973d3962e8c1423f01d0c3732fe5765cfca4290652ed83716e52e"} Jul 22 00:31:41.826716 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:41.826624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" event={"ID":"411b43ef-a9a2-4221-8057-db8435d41fea","Type":"ContainerStarted","Data":"a8be482f83eb58ab55c3f41dc5f3a1e7e9d7171cd53880d97d2d023b9979ed88"} Jul 22 00:31:41.828664 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:41.828619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44b7j" event={"ID":"c995ce90-2e54-4a9e-b74a-e579d7643edc","Type":"ContainerStarted","Data":"71f85dfd1d7f178cb9d4292ad84b61bfb847503352e48ff48c2e428046fb8d22"} Jul 22 00:31:41.842006 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:41.841945 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-frwrs" podStartSLOduration=5.401548495 podStartE2EDuration="8.841926364s" totalImagesPullingTime="3.367127467s" totalInitContainerRuntime="73.250402ms" isStatefulPod=true podCreationTimestamp="2026-07-22 00:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:41.840455615 +0000 UTC m=+286.470688161" watchObservedRunningTime="2026-07-22 00:31:41.841926364 +0000 UTC m=+286.472158886" Jul 22 00:31:44.841959 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.841922 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" event={"ID":"411b43ef-a9a2-4221-8057-db8435d41fea","Type":"ContainerStarted","Data":"24bb007b31e5dda9b4bbdd2d2539bc819fe03e561f3f26d782a9121f8a0f0940"} Jul 22 00:31:44.842376 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.842021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" event={"ID":"411b43ef-a9a2-4221-8057-db8435d41fea","Type":"ContainerStarted","Data":"978d3340c8efffc88d55b5f5cfc39baff80753a778a546f5237869a3bb32a6f7"} Jul 22 00:31:44.842376 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.842068 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:44.842376 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.842106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" event={"ID":"411b43ef-a9a2-4221-8057-db8435d41fea","Type":"ContainerStarted","Data":"a2ae2c3f3caf12c5ffda81531457b09cd1eba63a546cda6a1bf6dc869d8282bb"} Jul 22 00:31:44.843681 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.843657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44b7j" event={"ID":"c995ce90-2e54-4a9e-b74a-e579d7643edc","Type":"ContainerStarted","Data":"3efced66d2df03f2c770742490d6483393200a7a7ea7cf4697d1e80e2d9d9abc"} Jul 22 00:31:44.843808 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.843706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-44b7j" event={"ID":"c995ce90-2e54-4a9e-b74a-e579d7643edc","Type":"ContainerStarted","Data":"2d45a4fefa278672dff4af3faa634fb4150532baadec44ab340a60bfb5d94a3d"} Jul 22 00:31:44.843808 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.843732 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-44b7j" Jul 22 00:31:44.845144 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.845124 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857dbfbcbc-74ss7" event={"ID":"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66","Type":"ContainerStarted","Data":"2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb"} Jul 22 00:31:44.846504 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.846485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" event={"ID":"64a31bc3-e4e2-47ef-a515-c1bef899f354","Type":"ContainerStarted","Data":"8f48a55cc74b117828da7de1e84b36eb8d3c71cac75d138ffbb3347bdf3e5375"} Jul 22 00:31:44.846631 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.846609 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:44.848107 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.848078 2562 generic.go:356] "Generic (PLEG): container finished" podID="84396651-fb55-431a-a04b-6f5118d63d35" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" exitCode=0 Jul 22 00:31:44.848208 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.848124 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d"} Jul 22 00:31:44.851665 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.851631 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" Jul 22 00:31:44.863111 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.863068 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" podStartSLOduration=2.171597511 podStartE2EDuration="9.863055147s" totalImagesPullingTime="7.691457636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:44.861093218 +0000 UTC m=+289.491325758" watchObservedRunningTime="2026-07-22 00:31:44.863055147 +0000 UTC m=+289.493287686" Jul 22 00:31:44.875853 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.875812 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-44b7j" podStartSLOduration=251.452127722 podStartE2EDuration="4m14.875800793s" totalImagesPullingTime="3.423673071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:44.875045213 +0000 UTC m=+289.505277753" watchObservedRunningTime="2026-07-22 00:31:44.875800793 +0000 UTC m=+289.506033361" Jul 22 00:31:44.888991 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.888949 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-45wlh" podStartSLOduration=3.278034654 podStartE2EDuration="6.888934288s" totalImagesPullingTime="3.610899634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:44.887918709 +0000 UTC m=+289.518151248" watchObservedRunningTime="2026-07-22 00:31:44.888934288 +0000 UTC m=+289.519166847" Jul 22 00:31:44.929543 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:44.929496 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-857dbfbcbc-74ss7" podStartSLOduration=1.344384445 podStartE2EDuration="5.929479547s" totalImagesPullingTime="4.585095102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:44.928795715 +0000 UTC m=+289.559028256" watchObservedRunningTime="2026-07-22 00:31:44.929479547 +0000 UTC m=+289.559712086" Jul 22 00:31:45.857998 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:45.857971 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67f766cb47-dwg8j" Jul 22 00:31:46.925941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:46.925890 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" podUID="68d201d3-744d-4361-8589-f8b3429d5923" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 22 00:31:48.867137 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:48.867055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerStarted","Data":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} Jul 22 00:31:48.867137 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:48.867121 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerStarted","Data":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} Jul 22 00:31:48.867524 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:48.867157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerStarted","Data":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} Jul 22 00:31:48.867524 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:48.867181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerStarted","Data":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} Jul 22 00:31:48.867524 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:48.867208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerStarted","Data":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} Jul 22 00:31:48.867524 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:48.867237 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerStarted","Data":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} Jul 22 00:31:48.893892 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:48.893834 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.018424242 podStartE2EDuration="8.893814728s" totalImagesPullingTime="6.841732305s" totalInitContainerRuntime="33.658181ms" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:48.892745634 +0000 UTC m=+293.522978186" watchObservedRunningTime="2026-07-22 00:31:48.893814728 +0000 UTC m=+293.524047279" Jul 22 00:31:49.507678 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:49.507626 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:49.507829 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:49.507706 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:49.512296 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:49.512276 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:49.874988 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:49.874960 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:31:50.384962 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:50.384929 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:31:51.901483 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:31:51.901445 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x9f5n" podUID="88f41e66-c8b2-485f-809c-6e8f994c4541" Jul 22 00:31:54.854615 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:54.854587 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-44b7j" Jul 22 00:31:55.779059 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:55.779027 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:31:55.780318 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:55.780300 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:31:55.782632 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:55.782615 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 00:31:56.925612 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:56.925570 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" podUID="68d201d3-744d-4361-8589-f8b3429d5923" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 22 00:31:58.619795 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:58.619760 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:58.620175 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:58.619822 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:31:59.932716 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:59.932680 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44b7j_c995ce90-2e54-4a9e-b74a-e579d7643edc/dns/0.log" Jul 22 00:31:59.952276 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:31:59.952240 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44b7j_c995ce90-2e54-4a9e-b74a-e579d7643edc/kube-rbac-proxy/0.log" Jul 22 00:32:00.516583 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:00.516556 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jdphr_441030f4-d8a5-48cf-82d4-4882497d5560/dns-node-resolver/0.log" Jul 22 00:32:05.901733 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:05.901700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:32:06.925636 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:06.925598 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" podUID="68d201d3-744d-4361-8589-f8b3429d5923" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 22 00:32:06.926083 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:06.925691 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" Jul 22 00:32:06.926175 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:06.926147 2562 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2fba4562d8285658b4767b3de92cd05c854b87df07bfaca87317bdae2228b913"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 22 00:32:06.926223 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:06.926194 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" podUID="68d201d3-744d-4361-8589-f8b3429d5923" containerName="service-proxy" containerID="cri-o://2fba4562d8285658b4767b3de92cd05c854b87df07bfaca87317bdae2228b913" gracePeriod=30 Jul 22 00:32:07.938967 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:07.938930 2562 generic.go:356] "Generic (PLEG): container finished" podID="68d201d3-744d-4361-8589-f8b3429d5923" containerID="2fba4562d8285658b4767b3de92cd05c854b87df07bfaca87317bdae2228b913" exitCode=2 Jul 22 00:32:07.939357 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:07.938993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" event={"ID":"68d201d3-744d-4361-8589-f8b3429d5923","Type":"ContainerDied","Data":"2fba4562d8285658b4767b3de92cd05c854b87df07bfaca87317bdae2228b913"} Jul 22 00:32:07.939357 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:07.939037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59686576d5-vkwb9" event={"ID":"68d201d3-744d-4361-8589-f8b3429d5923","Type":"ContainerStarted","Data":"28a271d2b8e19d1b14c9e0c7fddacd32142ea405e363dc0013ca48600f86b28f"} Jul 22 00:32:18.624832 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:18.624802 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:32:18.628930 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:18.628905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69c588d6d6-kspc5" Jul 22 00:32:40.385191 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:40.385153 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:40.403960 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:40.403936 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:41.073821 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:41.073793 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:58.495419 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.495342 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:32:58.495944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.495628 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="prometheus" containerID="cri-o://a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" gracePeriod=600 Jul 22 00:32:58.495944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.495667 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="thanos-sidecar" containerID="cri-o://245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" gracePeriod=600 Jul 22 00:32:58.495944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.495667 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy" containerID="cri-o://5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" gracePeriod=600 Jul 22 00:32:58.495944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.495720 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-web" containerID="cri-o://78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" gracePeriod=600 Jul 22 00:32:58.495944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.495744 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-thanos" containerID="cri-o://86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" gracePeriod=600 Jul 22 00:32:58.495944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.495685 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="config-reloader" containerID="cri-o://152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" gracePeriod=600 Jul 22 00:32:58.729823 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.729801 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:58.780985 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.780905 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-tls-assets\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.780985 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.780938 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-metrics-client-ca\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.780985 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.780963 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-thanos-prometheus-http-client-file\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781243 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.780989 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-0\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781243 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781018 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-serving-certs-ca-bundle\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781243 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781051 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781243 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781079 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-config\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781243 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-tls\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781243 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781137 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-db\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781545 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781396 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-1\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781545 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781429 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:58.781545 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781441 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsr4v\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-kube-api-access-vsr4v\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781545 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781494 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781545 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781520 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-2\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781558 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-web-config\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781601 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-grpc-tls\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781631 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-metrics-client-certs\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781681 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-trusted-ca-bundle\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781710 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-config-out\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781739 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-kube-rbac-proxy\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.781810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.781766 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-kubelet-serving-ca-bundle\") pod \"84396651-fb55-431a-a04b-6f5118d63d35\" (UID: \"84396651-fb55-431a-a04b-6f5118d63d35\") " Jul 22 00:32:58.782136 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.782018 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-metrics-client-ca\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.782426 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.782380 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:32:58.782707 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.782672 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:58.782812 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.782758 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:58.783200 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.783131 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:58.783759 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.783632 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:58.783891 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.783851 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:58.784390 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.784364 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:58.784601 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.784565 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.784818 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.784786 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-config" (OuterVolumeSpecName: "config") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.784818 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.784801 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.785189 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.785050 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:58.785189 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.785166 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.786126 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.786099 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.786399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.786375 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-config-out" (OuterVolumeSpecName: "config-out") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:32:58.786771 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.786740 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.786854 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.786769 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.786854 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.786827 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.787009 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.786988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-kube-api-access-vsr4v" (OuterVolumeSpecName: "kube-api-access-vsr4v") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "kube-api-access-vsr4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:58.795568 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.795549 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-web-config" (OuterVolumeSpecName: "web-config") pod "84396651-fb55-431a-a04b-6f5118d63d35" (UID: "84396651-fb55-431a-a04b-6f5118d63d35"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:58.882864 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882829 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-config\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.882864 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882858 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.882864 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882869 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-db\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.882864 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882879 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882887 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr4v\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-kube-api-access-vsr4v\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882897 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882907 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882916 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-web-config\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882923 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-grpc-tls\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882931 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-metrics-client-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882939 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882947 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84396651-fb55-431a-a04b-6f5118d63d35-config-out\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882955 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-kube-rbac-proxy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882963 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882972 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84396651-fb55-431a-a04b-6f5118d63d35-tls-assets\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882980 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882988 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.882997 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84396651-fb55-431a-a04b-6f5118d63d35-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:58.883123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:58.883005 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84396651-fb55-431a-a04b-6f5118d63d35-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:32:59.122941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122908 2562 generic.go:356] "Generic (PLEG): container finished" podID="84396651-fb55-431a-a04b-6f5118d63d35" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" exitCode=0 Jul 22 00:32:59.122941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122933 2562 generic.go:356] "Generic (PLEG): container finished" podID="84396651-fb55-431a-a04b-6f5118d63d35" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" exitCode=0 Jul 22 00:32:59.122941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122941 2562 generic.go:356] "Generic (PLEG): container finished" podID="84396651-fb55-431a-a04b-6f5118d63d35" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" exitCode=0 Jul 22 00:32:59.122941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122948 2562 generic.go:356] "Generic (PLEG): container finished" podID="84396651-fb55-431a-a04b-6f5118d63d35" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" exitCode=0 Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122955 2562 generic.go:356] "Generic (PLEG): container finished" podID="84396651-fb55-431a-a04b-6f5118d63d35" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" exitCode=0 Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122962 2562 generic.go:356] "Generic (PLEG): container finished" podID="84396651-fb55-431a-a04b-6f5118d63d35" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" exitCode=0 Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122982 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.123050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.122992 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.123078 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.123103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.123126 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.123147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.123153 2562 scope.go:117] "RemoveContainer" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.123251 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.123170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84396651-fb55-431a-a04b-6f5118d63d35","Type":"ContainerDied","Data":"d1e6b94b29c0d68c6a068dfe7d403c37832a5733e458ea1f8c55427c1c5a4937"} Jul 22 00:32:59.131827 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.131811 2562 scope.go:117] "RemoveContainer" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.138541 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.138524 2562 scope.go:117] "RemoveContainer" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.145587 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.145566 2562 scope.go:117] "RemoveContainer" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.146046 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.146025 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:32:59.153471 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.151724 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:32:59.153661 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.153518 2562 scope.go:117] "RemoveContainer" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.160219 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.160204 2562 scope.go:117] "RemoveContainer" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.166843 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.166811 2562 scope.go:117] "RemoveContainer" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.172921 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.172907 2562 scope.go:117] "RemoveContainer" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.173166 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:32:59.173150 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": container with ID starting with 86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68 not found: ID does not exist" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.173212 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.173174 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} err="failed to get container status \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": rpc error: code = NotFound desc = could not find container \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": container with ID starting with 86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68 not found: ID does not exist" Jul 22 00:32:59.173212 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.173192 2562 scope.go:117] "RemoveContainer" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.173470 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:32:59.173442 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": container with ID starting with 5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318 not found: ID does not exist" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.173555 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.173500 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} err="failed to get container status \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": rpc error: code = NotFound desc = could not find container \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": container with ID starting with 5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318 not found: ID does not exist" Jul 22 00:32:59.173555 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.173523 2562 scope.go:117] "RemoveContainer" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.173832 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:32:59.173803 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": container with ID starting with 78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636 not found: ID does not exist" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.173895 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.173831 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} err="failed to get container status \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": rpc error: code = NotFound desc = could not find container \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": container with ID starting with 78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636 not found: ID does not exist" Jul 22 00:32:59.173895 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.173854 2562 scope.go:117] "RemoveContainer" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.174119 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:32:59.174094 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": container with ID starting with 245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27 not found: ID does not exist" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.174180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.174122 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} err="failed to get container status \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": rpc error: code = NotFound desc = could not find container \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": container with ID starting with 245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27 not found: ID does not exist" Jul 22 00:32:59.174180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.174139 2562 scope.go:117] "RemoveContainer" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.174507 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:32:59.174489 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": container with ID starting with 152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3 not found: ID does not exist" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.174572 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.174517 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} err="failed to get container status \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": rpc error: code = NotFound desc = could not find container \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": container with ID starting with 152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3 not found: ID does not exist" Jul 22 00:32:59.174572 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.174532 2562 scope.go:117] "RemoveContainer" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.174933 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:32:59.174913 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": container with ID starting with a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9 not found: ID does not exist" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.175014 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.174941 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} err="failed to get container status \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": rpc error: code = NotFound desc = could not find container \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": container with ID starting with a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9 not found: ID does not exist" Jul 22 00:32:59.175014 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.174962 2562 scope.go:117] "RemoveContainer" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.175133 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175117 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:32:59.175226 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:32:59.175208 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": container with ID starting with 06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d not found: ID does not exist" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.175261 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175234 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d"} err="failed to get container status \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": rpc error: code = NotFound desc = could not find container \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": container with ID starting with 06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d not found: ID does not exist" Jul 22 00:32:59.175261 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175255 2562 scope.go:117] "RemoveContainer" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.175422 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175410 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-thanos" Jul 22 00:32:59.175460 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175424 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-thanos" Jul 22 00:32:59.175460 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175440 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="init-config-reloader" Jul 22 00:32:59.175460 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175448 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="init-config-reloader" Jul 22 00:32:59.175460 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175458 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-web" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175463 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-web" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175477 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="config-reloader" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175483 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="config-reloader" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175491 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="prometheus" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175496 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="prometheus" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175501 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="thanos-sidecar" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175505 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="thanos-sidecar" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175513 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175518 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175456 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} err="failed to get container status \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": rpc error: code = NotFound desc = could not find container \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": container with ID starting with 86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68 not found: ID does not exist" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175543 2562 scope.go:117] "RemoveContainer" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175562 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="config-reloader" Jul 22 00:32:59.175570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175573 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-web" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175582 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="prometheus" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175588 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="thanos-sidecar" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175594 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175600 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84396651-fb55-431a-a04b-6f5118d63d35" containerName="kube-rbac-proxy-thanos" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175762 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} err="failed to get container status \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": rpc error: code = NotFound desc = could not find container \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": container with ID starting with 5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318 not found: ID does not exist" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175776 2562 scope.go:117] "RemoveContainer" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.175982 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} err="failed to get container status \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": rpc error: code = NotFound desc = could not find container \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": container with ID starting with 78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636 not found: ID does not exist" Jul 22 00:32:59.176052 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176003 2562 scope.go:117] "RemoveContainer" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.176330 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176205 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} err="failed to get container status \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": rpc error: code = NotFound desc = could not find container \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": container with ID starting with 245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27 not found: ID does not exist" Jul 22 00:32:59.176330 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176223 2562 scope.go:117] "RemoveContainer" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.176449 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176433 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} err="failed to get container status \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": rpc error: code = NotFound desc = could not find container \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": container with ID starting with 152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3 not found: ID does not exist" Jul 22 00:32:59.176484 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176451 2562 scope.go:117] "RemoveContainer" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.176668 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176628 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} err="failed to get container status \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": rpc error: code = NotFound desc = could not find container \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": container with ID starting with a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9 not found: ID does not exist" Jul 22 00:32:59.176747 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176670 2562 scope.go:117] "RemoveContainer" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.176866 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176851 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d"} err="failed to get container status \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": rpc error: code = NotFound desc = could not find container \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": container with ID starting with 06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d not found: ID does not exist" Jul 22 00:32:59.176909 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.176867 2562 scope.go:117] "RemoveContainer" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.177042 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177025 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} err="failed to get container status \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": rpc error: code = NotFound desc = could not find container \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": container with ID starting with 86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68 not found: ID does not exist" Jul 22 00:32:59.177099 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177053 2562 scope.go:117] "RemoveContainer" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.177256 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177239 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} err="failed to get container status \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": rpc error: code = NotFound desc = could not find container \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": container with ID starting with 5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318 not found: ID does not exist" Jul 22 00:32:59.177312 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177258 2562 scope.go:117] "RemoveContainer" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.177446 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177430 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} err="failed to get container status \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": rpc error: code = NotFound desc = could not find container \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": container with ID starting with 78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636 not found: ID does not exist" Jul 22 00:32:59.177502 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177448 2562 scope.go:117] "RemoveContainer" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.177709 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177692 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} err="failed to get container status \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": rpc error: code = NotFound desc = could not find container \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": container with ID starting with 245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27 not found: ID does not exist" Jul 22 00:32:59.177761 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177710 2562 scope.go:117] "RemoveContainer" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.177877 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177860 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} err="failed to get container status \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": rpc error: code = NotFound desc = could not find container \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": container with ID starting with 152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3 not found: ID does not exist" Jul 22 00:32:59.177918 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.177878 2562 scope.go:117] "RemoveContainer" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.178060 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178043 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} err="failed to get container status \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": rpc error: code = NotFound desc = could not find container \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": container with ID starting with a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9 not found: ID does not exist" Jul 22 00:32:59.178099 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178061 2562 scope.go:117] "RemoveContainer" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.178254 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178237 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d"} err="failed to get container status \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": rpc error: code = NotFound desc = could not find container \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": container with ID starting with 06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d not found: ID does not exist" Jul 22 00:32:59.178321 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178256 2562 scope.go:117] "RemoveContainer" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.178445 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178429 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} err="failed to get container status \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": rpc error: code = NotFound desc = could not find container \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": container with ID starting with 86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68 not found: ID does not exist" Jul 22 00:32:59.178492 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178445 2562 scope.go:117] "RemoveContainer" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.178612 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178596 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} err="failed to get container status \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": rpc error: code = NotFound desc = could not find container \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": container with ID starting with 5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318 not found: ID does not exist" Jul 22 00:32:59.178674 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178614 2562 scope.go:117] "RemoveContainer" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.178811 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178795 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} err="failed to get container status \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": rpc error: code = NotFound desc = could not find container \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": container with ID starting with 78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636 not found: ID does not exist" Jul 22 00:32:59.178853 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178811 2562 scope.go:117] "RemoveContainer" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.179007 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.178989 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} err="failed to get container status \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": rpc error: code = NotFound desc = could not find container \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": container with ID starting with 245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27 not found: ID does not exist" Jul 22 00:32:59.179065 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179009 2562 scope.go:117] "RemoveContainer" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.179214 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179193 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} err="failed to get container status \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": rpc error: code = NotFound desc = could not find container \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": container with ID starting with 152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3 not found: ID does not exist" Jul 22 00:32:59.179256 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179215 2562 scope.go:117] "RemoveContainer" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.179451 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179435 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} err="failed to get container status \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": rpc error: code = NotFound desc = could not find container \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": container with ID starting with a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9 not found: ID does not exist" Jul 22 00:32:59.179508 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179451 2562 scope.go:117] "RemoveContainer" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.179673 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179630 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d"} err="failed to get container status \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": rpc error: code = NotFound desc = could not find container \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": container with ID starting with 06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d not found: ID does not exist" Jul 22 00:32:59.179712 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179694 2562 scope.go:117] "RemoveContainer" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.179894 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179877 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} err="failed to get container status \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": rpc error: code = NotFound desc = could not find container \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": container with ID starting with 86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68 not found: ID does not exist" Jul 22 00:32:59.179940 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.179894 2562 scope.go:117] "RemoveContainer" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.180112 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180094 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} err="failed to get container status \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": rpc error: code = NotFound desc = could not find container \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": container with ID starting with 5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318 not found: ID does not exist" Jul 22 00:32:59.180148 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180113 2562 scope.go:117] "RemoveContainer" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.180297 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180282 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} err="failed to get container status \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": rpc error: code = NotFound desc = could not find container \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": container with ID starting with 78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636 not found: ID does not exist" Jul 22 00:32:59.180343 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180297 2562 scope.go:117] "RemoveContainer" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.180481 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180467 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} err="failed to get container status \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": rpc error: code = NotFound desc = could not find container \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": container with ID starting with 245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27 not found: ID does not exist" Jul 22 00:32:59.180527 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180481 2562 scope.go:117] "RemoveContainer" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.180699 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180685 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} err="failed to get container status \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": rpc error: code = NotFound desc = could not find container \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": container with ID starting with 152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3 not found: ID does not exist" Jul 22 00:32:59.180753 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180699 2562 scope.go:117] "RemoveContainer" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.180847 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180833 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.180900 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180881 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} err="failed to get container status \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": rpc error: code = NotFound desc = could not find container \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": container with ID starting with a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9 not found: ID does not exist" Jul 22 00:32:59.180935 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.180901 2562 scope.go:117] "RemoveContainer" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.181123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181097 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d"} err="failed to get container status \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": rpc error: code = NotFound desc = could not find container \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": container with ID starting with 06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d not found: ID does not exist" Jul 22 00:32:59.181123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181119 2562 scope.go:117] "RemoveContainer" containerID="86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68" Jul 22 00:32:59.181385 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181368 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68"} err="failed to get container status \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": rpc error: code = NotFound desc = could not find container \"86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68\": container with ID starting with 86a5ef7ab242c636adbfd85e24dceae85f02d4593c1f096a5c28b7723300ac68 not found: ID does not exist" Jul 22 00:32:59.181440 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181388 2562 scope.go:117] "RemoveContainer" containerID="5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318" Jul 22 00:32:59.181671 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181635 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318"} err="failed to get container status \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": rpc error: code = NotFound desc = could not find container \"5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318\": container with ID starting with 5421e6f2b8edc7b1218f11374a18c5e761423c7c4d8d260a0fb638347e50f318 not found: ID does not exist" Jul 22 00:32:59.181729 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181675 2562 scope.go:117] "RemoveContainer" containerID="78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636" Jul 22 00:32:59.181919 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181898 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636"} err="failed to get container status \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": rpc error: code = NotFound desc = could not find container \"78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636\": container with ID starting with 78546298ec53d11639f9c79398692b6ad4cd86f939af1774c75fb03ff56db636 not found: ID does not exist" Jul 22 00:32:59.181919 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.181919 2562 scope.go:117] "RemoveContainer" containerID="245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27" Jul 22 00:32:59.182141 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.182124 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27"} err="failed to get container status \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": rpc error: code = NotFound desc = could not find container \"245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27\": container with ID starting with 245f9a40a227d50aacc47a26a74b0f1d7ead229175e5614379e07a6bdbd73d27 not found: ID does not exist" Jul 22 00:32:59.182206 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.182143 2562 scope.go:117] "RemoveContainer" containerID="152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3" Jul 22 00:32:59.182384 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.182368 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3"} err="failed to get container status \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": rpc error: code = NotFound desc = could not find container \"152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3\": container with ID starting with 152d79182cc7997c755b5dff116ce24ec8a14d0f10bb4f8300a1508b8767a5b3 not found: ID does not exist" Jul 22 00:32:59.182431 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.182385 2562 scope.go:117] "RemoveContainer" containerID="a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9" Jul 22 00:32:59.182596 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.182578 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9"} err="failed to get container status \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": rpc error: code = NotFound desc = could not find container \"a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9\": container with ID starting with a798d3d77259e6852946a4facee9c4436970f56dc9214c8577653ed7ee01f9e9 not found: ID does not exist" Jul 22 00:32:59.182681 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.182597 2562 scope.go:117] "RemoveContainer" containerID="06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d" Jul 22 00:32:59.182873 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.182855 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d"} err="failed to get container status \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": rpc error: code = NotFound desc = could not find container \"06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d\": container with ID starting with 06329facc3b4656bc74e122014980adf1a4571a04aea1b4bbdd25f6d23ffd87d not found: ID does not exist" Jul 22 00:32:59.183043 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.183031 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 00:32:59.183105 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.183090 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 00:32:59.183186 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.183170 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 00:32:59.183245 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.183187 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-db7khhrrkfi27\"" Jul 22 00:32:59.183300 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.183287 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 00:32:59.183626 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.183614 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 00:32:59.183833 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.183821 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 00:32:59.184045 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.184017 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 00:32:59.184045 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.184008 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 00:32:59.184207 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.184076 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 00:32:59.184207 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.184135 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 00:32:59.184207 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.184137 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qtt9l\"" Jul 22 00:32:59.184348 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.184225 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 00:32:59.184348 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.184309 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 00:32:59.187158 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.187140 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 00:32:59.189892 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.189870 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 00:32:59.194167 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.194148 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:32:59.285869 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.285831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-config\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.285869 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.285874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.285891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.285956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.285995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286019 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-web-config\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6af90903-42d2-4580-aa55-cd028372c053-config-out\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286138 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286248 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6af90903-42d2-4580-aa55-cd028372c053-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286317 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286574 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286574 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.286574 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.286374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrqh\" (UniqueName: \"kubernetes.io/projected/6af90903-42d2-4580-aa55-cd028372c053-kube-api-access-csrqh\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.386975 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.386883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.386975 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.386927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.386975 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.386948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.386975 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.386974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrqh\" (UniqueName: \"kubernetes.io/projected/6af90903-42d2-4580-aa55-cd028372c053-kube-api-access-csrqh\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-config\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387126 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-web-config\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6af90903-42d2-4580-aa55-cd028372c053-config-out\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387285 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387313 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6af90903-42d2-4580-aa55-cd028372c053-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387636 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.387744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387730 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.388160 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.387938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.388236 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.388214 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.388625 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.388602 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.390399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.388996 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.390399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.390011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.390399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.390076 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6af90903-42d2-4580-aa55-cd028372c053-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.390399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.390089 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.390399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.390133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.390399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.390291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.391310 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.390757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.391310 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.390960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-config\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.391310 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.391061 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-web-config\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.392229 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.392212 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6af90903-42d2-4580-aa55-cd028372c053-config-out\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.392354 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.392337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6af90903-42d2-4580-aa55-cd028372c053-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.392541 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.392527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.392592 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.392576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.392866 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.392851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6af90903-42d2-4580-aa55-cd028372c053-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.395315 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.395298 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrqh\" (UniqueName: \"kubernetes.io/projected/6af90903-42d2-4580-aa55-cd028372c053-kube-api-access-csrqh\") pod \"prometheus-k8s-0\" (UID: \"6af90903-42d2-4580-aa55-cd028372c053\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.490576 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.490542 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:32:59.626329 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.626297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 00:32:59.629245 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:32:59.629218 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af90903_42d2_4580_aa55_cd028372c053.slice/crio-7b53cf77b9ad93867df19238d3192eb056895590207dcbe89cd0b04290129463 WatchSource:0}: Error finding container 7b53cf77b9ad93867df19238d3192eb056895590207dcbe89cd0b04290129463: Status 404 returned error can't find the container with id 7b53cf77b9ad93867df19238d3192eb056895590207dcbe89cd0b04290129463 Jul 22 00:32:59.904208 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:32:59.904136 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84396651-fb55-431a-a04b-6f5118d63d35" path="/var/lib/kubelet/pods/84396651-fb55-431a-a04b-6f5118d63d35/volumes" Jul 22 00:33:00.127289 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:00.127250 2562 generic.go:356] "Generic (PLEG): container finished" podID="6af90903-42d2-4580-aa55-cd028372c053" containerID="037674c7a8b693e2ea042410620280c2429ccf63bcfbd46cdfa6ca5af7e63c0e" exitCode=0 Jul 22 00:33:00.127450 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:00.127338 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerDied","Data":"037674c7a8b693e2ea042410620280c2429ccf63bcfbd46cdfa6ca5af7e63c0e"} Jul 22 00:33:00.127450 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:00.127402 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerStarted","Data":"7b53cf77b9ad93867df19238d3192eb056895590207dcbe89cd0b04290129463"} Jul 22 00:33:01.134835 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:01.134798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerStarted","Data":"697f846b999f57e12f322c0e686aca84c64430e444525da620cf93c61f296a81"} Jul 22 00:33:01.135179 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:01.134877 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerStarted","Data":"a8ffc5b169134cf795c708f5b3149a4eff59d5509cbf7bd42f01231bc0bbbd35"} Jul 22 00:33:01.135179 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:01.134916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerStarted","Data":"6274d91ba652631ba1ce4e20947152edb136254aa95a9b88528c140db1a2006d"} Jul 22 00:33:01.135179 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:01.134952 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerStarted","Data":"ff5460ca96d977be8c638224fc34c5386009429064d35cf9b1dfc71d6178100c"} Jul 22 00:33:01.135179 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:01.134987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerStarted","Data":"db167ea5592446f338e2d5d39482545d47c6896532436200dbaa726e2bb9237e"} Jul 22 00:33:01.135179 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:01.135019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6af90903-42d2-4580-aa55-cd028372c053","Type":"ContainerStarted","Data":"b3768a2914d3a6b6897bab00ef11826166aa66d9df7d5893d53d59e2de58776b"} Jul 22 00:33:01.163629 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:01.163585 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.146987951 podStartE2EDuration="2.163570111s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.58216ms" isStatefulPod=false podCreationTimestamp="2026-07-22 00:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:33:01.162391335 +0000 UTC m=+365.792623888" watchObservedRunningTime="2026-07-22 00:33:01.163570111 +0000 UTC m=+365.793802650" Jul 22 00:33:04.491241 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:04.491200 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:33:09.676682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:09.676627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:33:09.678929 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:09.678906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88f41e66-c8b2-485f-809c-6e8f994c4541-metrics-certs\") pod \"network-metrics-daemon-x9f5n\" (UID: \"88f41e66-c8b2-485f-809c-6e8f994c4541\") " pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:33:09.804845 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:09.804811 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dh9z6\"" Jul 22 00:33:09.813487 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:09.813455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9f5n" Jul 22 00:33:09.939826 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:09.939748 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9f5n"] Jul 22 00:33:09.942500 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:33:09.942473 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f41e66_c8b2_485f_809c_6e8f994c4541.slice/crio-cfe8566228c815ff49a178be5d383709038d8d7304bfed36a14f2b41d2638d26 WatchSource:0}: Error finding container cfe8566228c815ff49a178be5d383709038d8d7304bfed36a14f2b41d2638d26: Status 404 returned error can't find the container with id cfe8566228c815ff49a178be5d383709038d8d7304bfed36a14f2b41d2638d26 Jul 22 00:33:09.944324 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:09.944311 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:33:10.171151 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:10.171118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9f5n" event={"ID":"88f41e66-c8b2-485f-809c-6e8f994c4541","Type":"ContainerStarted","Data":"cfe8566228c815ff49a178be5d383709038d8d7304bfed36a14f2b41d2638d26"} Jul 22 00:33:12.179533 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:12.179450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9f5n" event={"ID":"88f41e66-c8b2-485f-809c-6e8f994c4541","Type":"ContainerStarted","Data":"20e9a2e32e48c6a939ae4cb45dc450ca116d2afa6d2335c14c4e7d4065873543"} Jul 22 00:33:12.179533 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:12.179503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9f5n" event={"ID":"88f41e66-c8b2-485f-809c-6e8f994c4541","Type":"ContainerStarted","Data":"de19808580f904546c3e628137064839a8676d711de0eb5d1122a1da686c9232"} Jul 22 00:33:12.195671 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:12.195608 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x9f5n" podStartSLOduration=375.252883262 podStartE2EDuration="6m17.195591735s" totalImagesPullingTime="1.942708473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:33:12.193849225 +0000 UTC m=+376.824081764" watchObservedRunningTime="2026-07-22 00:33:12.195591735 +0000 UTC m=+376.825824274" Jul 22 00:33:13.400718 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:13.400689 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857dbfbcbc-74ss7"] Jul 22 00:33:38.420115 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.420049 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-857dbfbcbc-74ss7" podUID="a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" containerName="console" containerID="cri-o://2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb" gracePeriod=15 Jul 22 00:33:38.663340 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.663317 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:33:38.826829 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.826795 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-oauth-config\") pod \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " Jul 22 00:33:38.826829 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.826829 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-oauth-serving-cert\") pod \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " Jul 22 00:33:38.827011 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.826857 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-config\") pod \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " Jul 22 00:33:38.827011 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.826887 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wcfc\" (UniqueName: \"kubernetes.io/projected/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-kube-api-access-8wcfc\") pod \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " Jul 22 00:33:38.827011 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.826911 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-trusted-ca-bundle\") pod \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " Jul 22 00:33:38.827011 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.826979 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-service-ca\") pod \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " Jul 22 00:33:38.827180 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.827053 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-serving-cert\") pod \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\" (UID: \"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66\") " Jul 22 00:33:38.827368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.827316 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-config" (OuterVolumeSpecName: "console-config") pod "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" (UID: "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:33:38.827368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.827276 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" (UID: "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:33:38.827543 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.827406 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" (UID: "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:33:38.827543 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.827443 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-service-ca" (OuterVolumeSpecName: "service-ca") pod "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" (UID: "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:33:38.829247 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.829219 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" (UID: "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:33:38.829247 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.829237 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-kube-api-access-8wcfc" (OuterVolumeSpecName: "kube-api-access-8wcfc") pod "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" (UID: "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66"). InnerVolumeSpecName "kube-api-access-8wcfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:33:38.829379 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.829246 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" (UID: "a6c5d22b-52b1-43a3-aba3-ddb4910b3b66"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:33:38.928514 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.928474 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-serving-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:33:38.928514 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.928509 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-oauth-config\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:33:38.928514 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.928520 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-oauth-serving-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:33:38.928514 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.928529 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-console-config\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:33:38.928777 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.928541 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wcfc\" (UniqueName: \"kubernetes.io/projected/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-kube-api-access-8wcfc\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:33:38.928777 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.928549 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-trusted-ca-bundle\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:33:38.928777 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:38.928557 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66-service-ca\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:33:39.280097 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.280013 2562 generic.go:356] "Generic (PLEG): container finished" podID="a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" containerID="2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb" exitCode=0 Jul 22 00:33:39.280097 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.280058 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857dbfbcbc-74ss7" Jul 22 00:33:39.280267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.280097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857dbfbcbc-74ss7" event={"ID":"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66","Type":"ContainerDied","Data":"2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb"} Jul 22 00:33:39.280267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.280146 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857dbfbcbc-74ss7" event={"ID":"a6c5d22b-52b1-43a3-aba3-ddb4910b3b66","Type":"ContainerDied","Data":"5e709f1f46b93b11a5ed7e2aa9d3a9dd8025350e31d03efdb6f7e6427ab77e21"} Jul 22 00:33:39.280267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.280165 2562 scope.go:117] "RemoveContainer" containerID="2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb" Jul 22 00:33:39.288567 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.288551 2562 scope.go:117] "RemoveContainer" containerID="2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb" Jul 22 00:33:39.288830 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:33:39.288812 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb\": container with ID starting with 2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb not found: ID does not exist" containerID="2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb" Jul 22 00:33:39.288893 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.288836 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb"} err="failed to get container status \"2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb\": rpc error: code = NotFound desc = could not find container \"2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb\": container with ID starting with 2768a10e235b049c9f2f03e87a49ef1af8b887d5e47ce00a77da9dda5d5544eb not found: ID does not exist" Jul 22 00:33:39.299281 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.299260 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857dbfbcbc-74ss7"] Jul 22 00:33:39.304724 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.304706 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-857dbfbcbc-74ss7"] Jul 22 00:33:39.908608 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:39.904727 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" path="/var/lib/kubelet/pods/a6c5d22b-52b1-43a3-aba3-ddb4910b3b66/volumes" Jul 22 00:33:59.491758 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:59.491715 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:33:59.507074 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:33:59.507045 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:34:00.372816 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:34:00.372789 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 00:36:24.347358 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.347325 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg"] Jul 22 00:36:24.347811 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.347603 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" containerName="console" Jul 22 00:36:24.347811 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.347613 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" containerName="console" Jul 22 00:36:24.347811 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.347672 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c5d22b-52b1-43a3-aba3-ddb4910b3b66" containerName="console" Jul 22 00:36:24.350562 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.350548 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.353816 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.353787 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 00:36:24.353816 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.353812 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 00:36:24.353998 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.353826 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 00:36:24.353998 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.353846 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 00:36:24.353998 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.353920 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:36:24.354238 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.354217 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j22h7\"" Jul 22 00:36:24.360966 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.360944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg"] Jul 22 00:36:24.391372 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.391351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e161a16f-e1b7-4a67-96ff-c073121d85d2-metrics-cert\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.391468 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.391391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e161a16f-e1b7-4a67-96ff-c073121d85d2-cert\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.391468 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.391450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfkw\" (UniqueName: \"kubernetes.io/projected/e161a16f-e1b7-4a67-96ff-c073121d85d2-kube-api-access-cjfkw\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.391571 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.391482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e161a16f-e1b7-4a67-96ff-c073121d85d2-manager-config\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.491839 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.491814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfkw\" (UniqueName: \"kubernetes.io/projected/e161a16f-e1b7-4a67-96ff-c073121d85d2-kube-api-access-cjfkw\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.491839 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.491841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e161a16f-e1b7-4a67-96ff-c073121d85d2-manager-config\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.492056 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.491869 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e161a16f-e1b7-4a67-96ff-c073121d85d2-metrics-cert\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.492056 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.491909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e161a16f-e1b7-4a67-96ff-c073121d85d2-cert\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.492507 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.492485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e161a16f-e1b7-4a67-96ff-c073121d85d2-manager-config\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.494365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.494346 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e161a16f-e1b7-4a67-96ff-c073121d85d2-metrics-cert\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.494365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.494360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e161a16f-e1b7-4a67-96ff-c073121d85d2-cert\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.499244 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.499211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfkw\" (UniqueName: \"kubernetes.io/projected/e161a16f-e1b7-4a67-96ff-c073121d85d2-kube-api-access-cjfkw\") pod \"lws-controller-manager-7bd6f87dc6-xzhvg\" (UID: \"e161a16f-e1b7-4a67-96ff-c073121d85d2\") " pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.660246 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.660162 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:24.781169 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.781140 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg"] Jul 22 00:36:24.784181 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:36:24.784152 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode161a16f_e1b7_4a67_96ff_c073121d85d2.slice/crio-44229a6d87f2c1ce069b6462b37ab7d803ea60ae6388e86d4af681018df5c473 WatchSource:0}: Error finding container 44229a6d87f2c1ce069b6462b37ab7d803ea60ae6388e86d4af681018df5c473: Status 404 returned error can't find the container with id 44229a6d87f2c1ce069b6462b37ab7d803ea60ae6388e86d4af681018df5c473 Jul 22 00:36:24.872408 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:24.872375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" event={"ID":"e161a16f-e1b7-4a67-96ff-c073121d85d2","Type":"ContainerStarted","Data":"44229a6d87f2c1ce069b6462b37ab7d803ea60ae6388e86d4af681018df5c473"} Jul 22 00:36:27.885603 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:27.885564 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" event={"ID":"e161a16f-e1b7-4a67-96ff-c073121d85d2","Type":"ContainerStarted","Data":"c2d140bb897ce2391f7202d5674387ac54c098d882bc7ad67b6c50f1859fc7d7"} Jul 22 00:36:27.886094 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:27.885718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:27.904692 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:27.904624 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" podStartSLOduration=1.268659939 podStartE2EDuration="3.904610568s" totalImagesPullingTime="2.635950629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:36:27.903854499 +0000 UTC m=+572.534087040" watchObservedRunningTime="2026-07-22 00:36:27.904610568 +0000 UTC m=+572.534843109" Jul 22 00:36:35.325098 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.325064 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz"] Jul 22 00:36:35.331328 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.331304 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.333455 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.333435 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 00:36:35.333570 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.333549 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 00:36:35.333680 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.333659 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 00:36:35.333760 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.333740 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 00:36:35.333854 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.333840 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8wt49\"" Jul 22 00:36:35.363270 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.363247 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz"] Jul 22 00:36:35.379133 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.379107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f94935d9-4b4b-41df-934f-4208f1ad6386-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.379242 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.379141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f94935d9-4b4b-41df-934f-4208f1ad6386-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.379242 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.379161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f94935d9-4b4b-41df-934f-4208f1ad6386-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.379242 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.379189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94935d9-4b4b-41df-934f-4208f1ad6386-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.379395 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.379263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644pq\" (UniqueName: \"kubernetes.io/projected/f94935d9-4b4b-41df-934f-4208f1ad6386-kube-api-access-644pq\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.480508 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.480477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644pq\" (UniqueName: \"kubernetes.io/projected/f94935d9-4b4b-41df-934f-4208f1ad6386-kube-api-access-644pq\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.480686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.480518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f94935d9-4b4b-41df-934f-4208f1ad6386-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.480686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.480543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f94935d9-4b4b-41df-934f-4208f1ad6386-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.480686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.480569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f94935d9-4b4b-41df-934f-4208f1ad6386-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.480686 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.480606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94935d9-4b4b-41df-934f-4208f1ad6386-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.481028 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.481008 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f94935d9-4b4b-41df-934f-4208f1ad6386-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.482799 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.482780 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94935d9-4b4b-41df-934f-4208f1ad6386-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.482928 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.482911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f94935d9-4b4b-41df-934f-4208f1ad6386-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.483100 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.483078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f94935d9-4b4b-41df-934f-4208f1ad6386-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.488467 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.488447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-644pq\" (UniqueName: \"kubernetes.io/projected/f94935d9-4b4b-41df-934f-4208f1ad6386-kube-api-access-644pq\") pod \"opendatahub-operator-controller-manager-6bb84b476f-j7rgz\" (UID: \"f94935d9-4b4b-41df-934f-4208f1ad6386\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.640719 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.640636 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:35.798610 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.798582 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz"] Jul 22 00:36:35.801127 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:36:35.801098 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94935d9_4b4b_41df_934f_4208f1ad6386.slice/crio-8fe040c90b33e0390b51c8adabfd9714279157078261ef4953f492ebba7213e9 WatchSource:0}: Error finding container 8fe040c90b33e0390b51c8adabfd9714279157078261ef4953f492ebba7213e9: Status 404 returned error can't find the container with id 8fe040c90b33e0390b51c8adabfd9714279157078261ef4953f492ebba7213e9 Jul 22 00:36:35.917005 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:35.916921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" event={"ID":"f94935d9-4b4b-41df-934f-4208f1ad6386","Type":"ContainerStarted","Data":"8fe040c90b33e0390b51c8adabfd9714279157078261ef4953f492ebba7213e9"} Jul 22 00:36:38.891232 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:38.891202 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bd6f87dc6-xzhvg" Jul 22 00:36:39.935263 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:39.935230 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" event={"ID":"f94935d9-4b4b-41df-934f-4208f1ad6386","Type":"ContainerStarted","Data":"27b9acff8aa06aaa976b9df99986af768097a21acf570ea32f07fab88590ff28"} Jul 22 00:36:40.940438 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:40.940357 2562 generic.go:356] "Generic (PLEG): container finished" podID="f94935d9-4b4b-41df-934f-4208f1ad6386" containerID="27b9acff8aa06aaa976b9df99986af768097a21acf570ea32f07fab88590ff28" exitCode=0 Jul 22 00:36:40.940438 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:40.940409 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" event={"ID":"f94935d9-4b4b-41df-934f-4208f1ad6386","Type":"ContainerDied","Data":"27b9acff8aa06aaa976b9df99986af768097a21acf570ea32f07fab88590ff28"} Jul 22 00:36:41.945805 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:41.945769 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" event={"ID":"f94935d9-4b4b-41df-934f-4208f1ad6386","Type":"ContainerStarted","Data":"d8ba317771c4251ee7c0ce614b753fff23aab9d749edf1821786b3dddbd51db1"} Jul 22 00:36:41.946197 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:41.945868 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:41.967466 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:41.967427 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" podStartSLOduration=1.859890791 podStartE2EDuration="6.967412143s" totalImagesPullingTime="4.102442476s" totalInitContainerRuntime="1.005078876s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:36:41.966431064 +0000 UTC m=+586.596663606" watchObservedRunningTime="2026-07-22 00:36:41.967412143 +0000 UTC m=+586.597644681" Jul 22 00:36:52.952035 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:52.952004 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-j7rgz" Jul 22 00:36:55.806137 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:55.806105 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:36:55.806741 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:36:55.806723 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:37:04.866044 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.866011 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz"] Jul 22 00:37:04.869282 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.869261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:04.871552 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.871526 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8wk9x\"" Jul 22 00:37:04.871778 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.871761 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 00:37:04.878150 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.878127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz"] Jul 22 00:37:04.882113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.882091 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss"] Jul 22 00:37:04.885102 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.885086 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:04.887435 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.887415 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-khmlr\"" Jul 22 00:37:04.887533 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.887451 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 00:37:04.894753 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.894729 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss"] Jul 22 00:37:04.916047 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.916026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4l4n\" (UniqueName: \"kubernetes.io/projected/6ddcc417-9097-475a-924a-456207977712-kube-api-access-t4l4n\") pod \"kserve-controller-manager-d6f4b65b8-rg8qz\" (UID: \"6ddcc417-9097-475a-924a-456207977712\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:04.916154 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.916066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddcc417-9097-475a-924a-456207977712-cert\") pod \"kserve-controller-manager-d6f4b65b8-rg8qz\" (UID: \"6ddcc417-9097-475a-924a-456207977712\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:04.916434 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.916413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zbbx\" (UniqueName: \"kubernetes.io/projected/776cedef-60e1-4cae-a066-545d90fd56a8-kube-api-access-2zbbx\") pod \"llmisvc-controller-manager-777c6bd7d5-8w8ss\" (UID: \"776cedef-60e1-4cae-a066-545d90fd56a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:04.916509 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:04.916446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/776cedef-60e1-4cae-a066-545d90fd56a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8w8ss\" (UID: \"776cedef-60e1-4cae-a066-545d90fd56a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:05.017162 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.017138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zbbx\" (UniqueName: \"kubernetes.io/projected/776cedef-60e1-4cae-a066-545d90fd56a8-kube-api-access-2zbbx\") pod \"llmisvc-controller-manager-777c6bd7d5-8w8ss\" (UID: \"776cedef-60e1-4cae-a066-545d90fd56a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:05.017294 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.017167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/776cedef-60e1-4cae-a066-545d90fd56a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8w8ss\" (UID: \"776cedef-60e1-4cae-a066-545d90fd56a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:05.017294 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.017193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4l4n\" (UniqueName: \"kubernetes.io/projected/6ddcc417-9097-475a-924a-456207977712-kube-api-access-t4l4n\") pod \"kserve-controller-manager-d6f4b65b8-rg8qz\" (UID: \"6ddcc417-9097-475a-924a-456207977712\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:05.017387 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:37:05.017290 2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 22 00:37:05.017387 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.017332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddcc417-9097-475a-924a-456207977712-cert\") pod \"kserve-controller-manager-d6f4b65b8-rg8qz\" (UID: \"6ddcc417-9097-475a-924a-456207977712\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:05.017387 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:37:05.017368 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/776cedef-60e1-4cae-a066-545d90fd56a8-cert podName:776cedef-60e1-4cae-a066-545d90fd56a8 nodeName:}" failed. No retries permitted until 2026-07-22 00:37:05.517349457 +0000 UTC m=+610.147581979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/776cedef-60e1-4cae-a066-545d90fd56a8-cert") pod "llmisvc-controller-manager-777c6bd7d5-8w8ss" (UID: "776cedef-60e1-4cae-a066-545d90fd56a8") : secret "llmisvc-webhook-server-cert" not found Jul 22 00:37:05.017482 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:37:05.017402 2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 22 00:37:05.017482 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:37:05.017462 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ddcc417-9097-475a-924a-456207977712-cert podName:6ddcc417-9097-475a-924a-456207977712 nodeName:}" failed. No retries permitted until 2026-07-22 00:37:05.517445759 +0000 UTC m=+610.147678295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ddcc417-9097-475a-924a-456207977712-cert") pod "kserve-controller-manager-d6f4b65b8-rg8qz" (UID: "6ddcc417-9097-475a-924a-456207977712") : secret "kserve-webhook-server-cert" not found Jul 22 00:37:05.035968 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.035943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zbbx\" (UniqueName: \"kubernetes.io/projected/776cedef-60e1-4cae-a066-545d90fd56a8-kube-api-access-2zbbx\") pod \"llmisvc-controller-manager-777c6bd7d5-8w8ss\" (UID: \"776cedef-60e1-4cae-a066-545d90fd56a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:05.037244 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.037223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4l4n\" (UniqueName: \"kubernetes.io/projected/6ddcc417-9097-475a-924a-456207977712-kube-api-access-t4l4n\") pod \"kserve-controller-manager-d6f4b65b8-rg8qz\" (UID: \"6ddcc417-9097-475a-924a-456207977712\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:05.521638 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.521598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/776cedef-60e1-4cae-a066-545d90fd56a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8w8ss\" (UID: \"776cedef-60e1-4cae-a066-545d90fd56a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:05.521882 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.521682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddcc417-9097-475a-924a-456207977712-cert\") pod \"kserve-controller-manager-d6f4b65b8-rg8qz\" (UID: \"6ddcc417-9097-475a-924a-456207977712\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:05.523869 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.523845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddcc417-9097-475a-924a-456207977712-cert\") pod \"kserve-controller-manager-d6f4b65b8-rg8qz\" (UID: \"6ddcc417-9097-475a-924a-456207977712\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:05.523980 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.523878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/776cedef-60e1-4cae-a066-545d90fd56a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8w8ss\" (UID: \"776cedef-60e1-4cae-a066-545d90fd56a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:05.779449 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.779368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:05.795284 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.795263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:05.919160 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.919121 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz"] Jul 22 00:37:05.921585 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:37:05.921553 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddcc417_9097_475a_924a_456207977712.slice/crio-695d6a9c6b572ac921ed4f668c105460462d4d3441b203e569cd061ee6e80889 WatchSource:0}: Error finding container 695d6a9c6b572ac921ed4f668c105460462d4d3441b203e569cd061ee6e80889: Status 404 returned error can't find the container with id 695d6a9c6b572ac921ed4f668c105460462d4d3441b203e569cd061ee6e80889 Jul 22 00:37:05.939097 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:05.939074 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss"] Jul 22 00:37:05.942459 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:37:05.942434 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod776cedef_60e1_4cae_a066_545d90fd56a8.slice/crio-ee53067d1e0ed988e734787bdc1a21cd7d04082a9233abfd96c476c64cf3eddd WatchSource:0}: Error finding container ee53067d1e0ed988e734787bdc1a21cd7d04082a9233abfd96c476c64cf3eddd: Status 404 returned error can't find the container with id ee53067d1e0ed988e734787bdc1a21cd7d04082a9233abfd96c476c64cf3eddd Jul 22 00:37:06.039057 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:06.038984 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" event={"ID":"6ddcc417-9097-475a-924a-456207977712","Type":"ContainerStarted","Data":"695d6a9c6b572ac921ed4f668c105460462d4d3441b203e569cd061ee6e80889"} Jul 22 00:37:06.040091 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:06.040069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" event={"ID":"776cedef-60e1-4cae-a066-545d90fd56a8","Type":"ContainerStarted","Data":"ee53067d1e0ed988e734787bdc1a21cd7d04082a9233abfd96c476c64cf3eddd"} Jul 22 00:37:08.012247 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.012210 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl"] Jul 22 00:37:08.015836 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.015814 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.020536 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.020503 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 00:37:08.020679 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.020546 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 00:37:08.020994 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.020958 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 00:37:08.022126 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.022073 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dm25k\"" Jul 22 00:37:08.028721 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.028697 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl"] Jul 22 00:37:08.041741 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.041704 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r79rh\" (UniqueName: \"kubernetes.io/projected/174e6bc7-959e-4306-ba1e-7d5d06bdb080-kube-api-access-r79rh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.041844 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.041798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.041844 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.041828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.041961 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.041869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.041961 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.041916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.042253 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.041977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.042253 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.042028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.042253 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.042075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.042253 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.042109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.142817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.142785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.142817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.142822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143051 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.142844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143051 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.142879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143209 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r79rh\" (UniqueName: \"kubernetes.io/projected/174e6bc7-959e-4306-ba1e-7d5d06bdb080-kube-api-access-r79rh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143350 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143350 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143350 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143507 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143507 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143378 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143507 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143793 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143896 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143840 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.143990 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.143957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.145481 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.145463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.146006 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.145984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.151720 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.151701 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174e6bc7-959e-4306-ba1e-7d5d06bdb080-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.151848 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.151825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r79rh\" (UniqueName: \"kubernetes.io/projected/174e6bc7-959e-4306-ba1e-7d5d06bdb080-kube-api-access-r79rh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hgmfl\" (UID: \"174e6bc7-959e-4306-ba1e-7d5d06bdb080\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:08.330195 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:08.330161 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:09.520799 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:09.520768 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl"] Jul 22 00:37:09.523450 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:37:09.523424 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174e6bc7_959e_4306_ba1e_7d5d06bdb080.slice/crio-f789af2c5147b8261208cccfaccd6b83770d2672e4b61c154279b97c8882e1ab WatchSource:0}: Error finding container f789af2c5147b8261208cccfaccd6b83770d2672e4b61c154279b97c8882e1ab: Status 404 returned error can't find the container with id f789af2c5147b8261208cccfaccd6b83770d2672e4b61c154279b97c8882e1ab Jul 22 00:37:10.062810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:10.062770 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" event={"ID":"174e6bc7-959e-4306-ba1e-7d5d06bdb080","Type":"ContainerStarted","Data":"f789af2c5147b8261208cccfaccd6b83770d2672e4b61c154279b97c8882e1ab"} Jul 22 00:37:10.066909 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:10.066882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" event={"ID":"776cedef-60e1-4cae-a066-545d90fd56a8","Type":"ContainerStarted","Data":"42c10586212618af62df970046a4f018941f8171558ab6a11e7e818affa3b285"} Jul 22 00:37:10.067057 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:10.066936 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:10.068376 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:10.068356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" event={"ID":"6ddcc417-9097-475a-924a-456207977712","Type":"ContainerStarted","Data":"1caec828e5c7e1322d028bfe5aa63a69b5f2a78be9d9ae0a3e1c4ffbdc23b0b1"} Jul 22 00:37:10.068482 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:10.068472 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:37:10.083006 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:10.082967 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" podStartSLOduration=2.627520337 podStartE2EDuration="6.08295474s" totalImagesPullingTime="3.455434403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:10.081052766 +0000 UTC m=+614.711285305" watchObservedRunningTime="2026-07-22 00:37:10.08295474 +0000 UTC m=+614.713187343" Jul 22 00:37:10.096393 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:10.096356 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" podStartSLOduration=2.624080797 podStartE2EDuration="6.0963446s" totalImagesPullingTime="3.472263803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:10.095193067 +0000 UTC m=+614.725425603" watchObservedRunningTime="2026-07-22 00:37:10.0963446 +0000 UTC m=+614.726577142" Jul 22 00:37:12.607512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:12.607478 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:37:12.607777 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:12.607554 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:37:12.607777 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:12.607582 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:37:13.083998 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:13.083968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" event={"ID":"174e6bc7-959e-4306-ba1e-7d5d06bdb080","Type":"ContainerStarted","Data":"430b31502bc656aa0e1a39713e815a5cb69a5bd5ef9f24d1df580e8247e5f582"} Jul 22 00:37:13.105467 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:13.105423 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" podStartSLOduration=3.023692461 podStartE2EDuration="6.1054087s" totalImagesPullingTime="3.081716239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:13.104359525 +0000 UTC m=+617.734592091" watchObservedRunningTime="2026-07-22 00:37:13.1054087 +0000 UTC m=+617.735641240" Jul 22 00:37:13.330726 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:13.330685 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:13.334899 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:13.334845 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:14.088423 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:14.088393 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:14.089428 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:14.089409 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hgmfl" Jul 22 00:37:41.075474 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:41.075400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8w8ss" Jul 22 00:37:41.078588 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:37:41.078569 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rg8qz" Jul 22 00:38:28.740718 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.740683 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf"] Jul 22 00:38:28.742991 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.742974 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" Jul 22 00:38:28.745034 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.745012 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 00:38:28.745144 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.745102 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 00:38:28.745213 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.745200 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 22 00:38:28.745842 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.745822 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s22pw\"" Jul 22 00:38:28.754869 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.754846 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf"] Jul 22 00:38:28.834944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.834917 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcb7\" (UniqueName: \"kubernetes.io/projected/7c568dff-b531-417d-8393-8abf64065029-kube-api-access-pbcb7\") pod \"dns-operator-controller-manager-864dd78c4c-xxsmf\" (UID: \"7c568dff-b531-417d-8393-8abf64065029\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" Jul 22 00:38:28.935591 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.935543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcb7\" (UniqueName: \"kubernetes.io/projected/7c568dff-b531-417d-8393-8abf64065029-kube-api-access-pbcb7\") pod \"dns-operator-controller-manager-864dd78c4c-xxsmf\" (UID: \"7c568dff-b531-417d-8393-8abf64065029\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" Jul 22 00:38:28.943699 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:28.943671 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcb7\" (UniqueName: \"kubernetes.io/projected/7c568dff-b531-417d-8393-8abf64065029-kube-api-access-pbcb7\") pod \"dns-operator-controller-manager-864dd78c4c-xxsmf\" (UID: \"7c568dff-b531-417d-8393-8abf64065029\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" Jul 22 00:38:29.052257 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:29.052228 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" Jul 22 00:38:29.180510 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:29.180478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf"] Jul 22 00:38:29.183153 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:38:29.183129 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c568dff_b531_417d_8393_8abf64065029.slice/crio-292d8bc4d0ae84f1cad7360c343e1e17867e1363d98dbfc988d748439f11598d WatchSource:0}: Error finding container 292d8bc4d0ae84f1cad7360c343e1e17867e1363d98dbfc988d748439f11598d: Status 404 returned error can't find the container with id 292d8bc4d0ae84f1cad7360c343e1e17867e1363d98dbfc988d748439f11598d Jul 22 00:38:29.185099 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:29.185079 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:38:29.402105 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:29.402022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" event={"ID":"7c568dff-b531-417d-8393-8abf64065029","Type":"ContainerStarted","Data":"292d8bc4d0ae84f1cad7360c343e1e17867e1363d98dbfc988d748439f11598d"} Jul 22 00:38:32.421356 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:32.421298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" event={"ID":"7c568dff-b531-417d-8393-8abf64065029","Type":"ContainerStarted","Data":"8dbb28ee6a4e8b3fe82286321bc5e84e7baa92edd20f84503f536909b20381fd"} Jul 22 00:38:32.421811 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:32.421417 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" Jul 22 00:38:32.444031 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:32.443978 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" podStartSLOduration=2.236898541 podStartE2EDuration="4.443962224s" totalImagesPullingTime="2.207063683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:32.442595475 +0000 UTC m=+697.072828016" watchObservedRunningTime="2026-07-22 00:38:32.443962224 +0000 UTC m=+697.074194763" Jul 22 00:38:37.528819 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.528778 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-52w5l"] Jul 22 00:38:37.532182 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.532163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" Jul 22 00:38:37.534609 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.534591 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nd9vx\"" Jul 22 00:38:37.542030 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.542008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-52w5l"] Jul 22 00:38:37.711592 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.711554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd69q\" (UniqueName: \"kubernetes.io/projected/e59ba304-594e-45a4-b49f-5c1929122242-kube-api-access-sd69q\") pod \"authorino-operator-75cc8d5cb7-52w5l\" (UID: \"e59ba304-594e-45a4-b49f-5c1929122242\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" Jul 22 00:38:37.812687 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.812636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd69q\" (UniqueName: \"kubernetes.io/projected/e59ba304-594e-45a4-b49f-5c1929122242-kube-api-access-sd69q\") pod \"authorino-operator-75cc8d5cb7-52w5l\" (UID: \"e59ba304-594e-45a4-b49f-5c1929122242\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" Jul 22 00:38:37.824348 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.824322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd69q\" (UniqueName: \"kubernetes.io/projected/e59ba304-594e-45a4-b49f-5c1929122242-kube-api-access-sd69q\") pod \"authorino-operator-75cc8d5cb7-52w5l\" (UID: \"e59ba304-594e-45a4-b49f-5c1929122242\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" Jul 22 00:38:37.842288 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.842260 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" Jul 22 00:38:37.969147 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:37.969116 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-52w5l"] Jul 22 00:38:37.972035 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:38:37.972008 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59ba304_594e_45a4_b49f_5c1929122242.slice/crio-c82ee07e659adc33bb529106f2ada0c2dd5bfee78f0e8fdb4d7ad96af036381e WatchSource:0}: Error finding container c82ee07e659adc33bb529106f2ada0c2dd5bfee78f0e8fdb4d7ad96af036381e: Status 404 returned error can't find the container with id c82ee07e659adc33bb529106f2ada0c2dd5bfee78f0e8fdb4d7ad96af036381e Jul 22 00:38:38.450858 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:38.450827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" event={"ID":"e59ba304-594e-45a4-b49f-5c1929122242","Type":"ContainerStarted","Data":"c82ee07e659adc33bb529106f2ada0c2dd5bfee78f0e8fdb4d7ad96af036381e"} Jul 22 00:38:41.466272 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:41.466238 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" event={"ID":"e59ba304-594e-45a4-b49f-5c1929122242","Type":"ContainerStarted","Data":"58cb188db415419b3059270a3c8c93fb79dd28cc8586611de48cf42d59311ee4"} Jul 22 00:38:41.466696 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:41.466399 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" Jul 22 00:38:41.486090 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:41.486045 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" podStartSLOduration=1.132662509 podStartE2EDuration="4.486030514s" totalImagesPullingTime="3.353368005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:41.484435488 +0000 UTC m=+706.114668041" watchObservedRunningTime="2026-07-22 00:38:41.486030514 +0000 UTC m=+706.116263055" Jul 22 00:38:43.428384 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:43.428355 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xxsmf" Jul 22 00:38:52.473453 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:52.473351 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-52w5l" Jul 22 00:38:53.041653 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.041607 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz"] Jul 22 00:38:53.044525 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.044504 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.046944 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.046920 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pr929\"" Jul 22 00:38:53.068021 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.067997 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz"] Jul 22 00:38:53.139943 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.139910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7148c80d-9448-4e12-b7e5-3907e4898ec0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.140081 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.139982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsj8w\" (UniqueName: \"kubernetes.io/projected/7148c80d-9448-4e12-b7e5-3907e4898ec0-kube-api-access-dsj8w\") pod \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.240855 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.240820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsj8w\" (UniqueName: \"kubernetes.io/projected/7148c80d-9448-4e12-b7e5-3907e4898ec0-kube-api-access-dsj8w\") pod \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.240997 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.240874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7148c80d-9448-4e12-b7e5-3907e4898ec0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.241231 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.241216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7148c80d-9448-4e12-b7e5-3907e4898ec0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.256512 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.256480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsj8w\" (UniqueName: \"kubernetes.io/projected/7148c80d-9448-4e12-b7e5-3907e4898ec0-kube-api-access-dsj8w\") pod \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.354797 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.354719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:53.484310 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.484280 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz"] Jul 22 00:38:53.487678 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:38:53.487636 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7148c80d_9448_4e12_b7e5_3907e4898ec0.slice/crio-f0093ec5218d3a27bb699ae98e090cd18d184c1a05e714ad4398d6277123a0a6 WatchSource:0}: Error finding container f0093ec5218d3a27bb699ae98e090cd18d184c1a05e714ad4398d6277123a0a6: Status 404 returned error can't find the container with id f0093ec5218d3a27bb699ae98e090cd18d184c1a05e714ad4398d6277123a0a6 Jul 22 00:38:53.522126 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.522101 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz"] Jul 22 00:38:53.525408 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.525383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" event={"ID":"7148c80d-9448-4e12-b7e5-3907e4898ec0","Type":"ContainerStarted","Data":"f0093ec5218d3a27bb699ae98e090cd18d184c1a05e714ad4398d6277123a0a6"} Jul 22 00:38:53.528880 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.528858 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz"] Jul 22 00:38:53.550094 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.550072 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5"] Jul 22 00:38:53.554012 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.553997 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.567883 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.567860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5"] Jul 22 00:38:53.643820 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.643728 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rwsw5\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.643820 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.643795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdk5\" (UniqueName: \"kubernetes.io/projected/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-kube-api-access-dmdk5\") pod \"kuadrant-operator-controller-manager-55467d9f7-rwsw5\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.744280 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.744243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdk5\" (UniqueName: \"kubernetes.io/projected/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-kube-api-access-dmdk5\") pod \"kuadrant-operator-controller-manager-55467d9f7-rwsw5\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.744462 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.744310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rwsw5\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.744685 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.744668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-rwsw5\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.751704 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.751680 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdk5\" (UniqueName: \"kubernetes.io/projected/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-kube-api-access-dmdk5\") pod \"kuadrant-operator-controller-manager-55467d9f7-rwsw5\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.864533 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.864490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:53.991083 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:53.991058 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5"] Jul 22 00:38:53.993621 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:38:53.993600 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83bd0fd_9fef_4b3c_9403_3782dc5b0956.slice/crio-ec2045a6f33beb13953cfc23b6a99d3eb8d4b50856988cffd4028cb3df22357e WatchSource:0}: Error finding container ec2045a6f33beb13953cfc23b6a99d3eb8d4b50856988cffd4028cb3df22357e: Status 404 returned error can't find the container with id ec2045a6f33beb13953cfc23b6a99d3eb8d4b50856988cffd4028cb3df22357e Jul 22 00:38:54.532060 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:54.532027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" event={"ID":"c83bd0fd-9fef-4b3c-9403-3782dc5b0956","Type":"ContainerStarted","Data":"ec2045a6f33beb13953cfc23b6a99d3eb8d4b50856988cffd4028cb3df22357e"} Jul 22 00:38:58.555107 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.555069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" event={"ID":"c83bd0fd-9fef-4b3c-9403-3782dc5b0956","Type":"ContainerStarted","Data":"cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0"} Jul 22 00:38:58.555526 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.555131 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:38:58.556748 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.556721 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" containerName="manager" containerID="cri-o://4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35" gracePeriod=2 Jul 22 00:38:58.574518 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.574461 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" podStartSLOduration=1.448177249 podStartE2EDuration="5.574441955s" totalImagesPullingTime="4.126264706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:58.572537722 +0000 UTC m=+723.202770259" watchObservedRunningTime="2026-07-22 00:38:58.574441955 +0000 UTC m=+723.204674498" Jul 22 00:38:58.574687 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:38:58.574661 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:58.796482 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.796458 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:58.798406 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:38:58.798378 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:58.892150 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.892060 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7148c80d-9448-4e12-b7e5-3907e4898ec0-extensions-socket-volume\") pod \"7148c80d-9448-4e12-b7e5-3907e4898ec0\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " Jul 22 00:38:58.892150 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.892102 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsj8w\" (UniqueName: \"kubernetes.io/projected/7148c80d-9448-4e12-b7e5-3907e4898ec0-kube-api-access-dsj8w\") pod \"7148c80d-9448-4e12-b7e5-3907e4898ec0\" (UID: \"7148c80d-9448-4e12-b7e5-3907e4898ec0\") " Jul 22 00:38:58.892377 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.892349 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7148c80d-9448-4e12-b7e5-3907e4898ec0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7148c80d-9448-4e12-b7e5-3907e4898ec0" (UID: "7148c80d-9448-4e12-b7e5-3907e4898ec0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:58.894211 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.894191 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7148c80d-9448-4e12-b7e5-3907e4898ec0-kube-api-access-dsj8w" (OuterVolumeSpecName: "kube-api-access-dsj8w") pod "7148c80d-9448-4e12-b7e5-3907e4898ec0" (UID: "7148c80d-9448-4e12-b7e5-3907e4898ec0"). InnerVolumeSpecName "kube-api-access-dsj8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:58.992853 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.992820 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7148c80d-9448-4e12-b7e5-3907e4898ec0-extensions-socket-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:38:58.992853 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:58.992845 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsj8w\" (UniqueName: \"kubernetes.io/projected/7148c80d-9448-4e12-b7e5-3907e4898ec0-kube-api-access-dsj8w\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:38:59.562590 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:59.562543 2562 generic.go:356] "Generic (PLEG): container finished" podID="7148c80d-9448-4e12-b7e5-3907e4898ec0" containerID="4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35" exitCode=2 Jul 22 00:38:59.562590 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:59.562593 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:59.563113 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:59.562665 2562 scope.go:117] "RemoveContainer" containerID="4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35" Jul 22 00:38:59.564785 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:38:59.564752 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:59.571061 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:59.571046 2562 scope.go:117] "RemoveContainer" containerID="4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35" Jul 22 00:38:59.571311 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:38:59.571295 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35\": container with ID starting with 4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35 not found: ID does not exist" containerID="4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35" Jul 22 00:38:59.571352 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:59.571321 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35"} err="failed to get container status \"4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35\": rpc error: code = NotFound desc = could not find container \"4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35\": container with ID starting with 4f3d363c92b380c954fb1345de53867531623d630cc57570129fbfd839b18c35 not found: ID does not exist" Jul 22 00:38:59.572249 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:38:59.572230 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5vqsz\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5vqsz" Jul 22 00:38:59.905161 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:38:59.905084 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" path="/var/lib/kubelet/pods/7148c80d-9448-4e12-b7e5-3907e4898ec0/volumes" Jul 22 00:39:09.565129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:09.565096 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:39:21.993776 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:21.993744 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5"] Jul 22 00:39:21.994212 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:21.993976 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" podUID="c83bd0fd-9fef-4b3c-9403-3782dc5b0956" containerName="manager" containerID="cri-o://cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0" gracePeriod=10 Jul 22 00:39:22.245754 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.245699 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:39:22.394932 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.394900 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-extensions-socket-volume\") pod \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " Jul 22 00:39:22.395093 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.395026 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmdk5\" (UniqueName: \"kubernetes.io/projected/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-kube-api-access-dmdk5\") pod \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\" (UID: \"c83bd0fd-9fef-4b3c-9403-3782dc5b0956\") " Jul 22 00:39:22.395411 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.395391 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c83bd0fd-9fef-4b3c-9403-3782dc5b0956" (UID: "c83bd0fd-9fef-4b3c-9403-3782dc5b0956"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:39:22.397127 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.397105 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-kube-api-access-dmdk5" (OuterVolumeSpecName: "kube-api-access-dmdk5") pod "c83bd0fd-9fef-4b3c-9403-3782dc5b0956" (UID: "c83bd0fd-9fef-4b3c-9403-3782dc5b0956"). InnerVolumeSpecName "kube-api-access-dmdk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:39:22.496054 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.495980 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-extensions-socket-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:39:22.496054 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.496007 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdk5\" (UniqueName: \"kubernetes.io/projected/c83bd0fd-9fef-4b3c-9403-3782dc5b0956-kube-api-access-dmdk5\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:39:22.664176 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.664145 2562 generic.go:356] "Generic (PLEG): container finished" podID="c83bd0fd-9fef-4b3c-9403-3782dc5b0956" containerID="cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0" exitCode=0 Jul 22 00:39:22.664356 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.664189 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" Jul 22 00:39:22.664356 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.664231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" event={"ID":"c83bd0fd-9fef-4b3c-9403-3782dc5b0956","Type":"ContainerDied","Data":"cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0"} Jul 22 00:39:22.664356 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.664286 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5" event={"ID":"c83bd0fd-9fef-4b3c-9403-3782dc5b0956","Type":"ContainerDied","Data":"ec2045a6f33beb13953cfc23b6a99d3eb8d4b50856988cffd4028cb3df22357e"} Jul 22 00:39:22.664356 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.664314 2562 scope.go:117] "RemoveContainer" containerID="cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0" Jul 22 00:39:22.673275 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.673260 2562 scope.go:117] "RemoveContainer" containerID="cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0" Jul 22 00:39:22.673552 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:39:22.673534 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0\": container with ID starting with cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0 not found: ID does not exist" containerID="cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0" Jul 22 00:39:22.673602 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.673558 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0"} err="failed to get container status \"cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0\": rpc error: code = NotFound desc = could not find container \"cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0\": container with ID starting with cfe852d51d45b87a6fa39a03f89fad17a95567473655e382383ed487357f38f0 not found: ID does not exist" Jul 22 00:39:22.684255 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.684231 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5"] Jul 22 00:39:22.689403 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:22.689365 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-rwsw5"] Jul 22 00:39:23.904624 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:23.904591 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83bd0fd-9fef-4b3c-9403-3782dc5b0956" path="/var/lib/kubelet/pods/c83bd0fd-9fef-4b3c-9403-3782dc5b0956/volumes" Jul 22 00:39:25.971739 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.971710 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw"] Jul 22 00:39:25.972195 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.972044 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" containerName="manager" Jul 22 00:39:25.972195 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.972055 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" containerName="manager" Jul 22 00:39:25.972195 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.972081 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83bd0fd-9fef-4b3c-9403-3782dc5b0956" containerName="manager" Jul 22 00:39:25.972195 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.972086 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83bd0fd-9fef-4b3c-9403-3782dc5b0956" containerName="manager" Jul 22 00:39:25.972195 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.972137 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7148c80d-9448-4e12-b7e5-3907e4898ec0" containerName="manager" Jul 22 00:39:25.972195 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.972145 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83bd0fd-9fef-4b3c-9403-3782dc5b0956" containerName="manager" Jul 22 00:39:25.974917 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.974897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:25.977264 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.977248 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x5sn5\"" Jul 22 00:39:25.985789 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:25.985768 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw"] Jul 22 00:39:26.124704 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.124704 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smpf\" (UniqueName: \"kubernetes.io/projected/60abed7c-dc00-409b-9613-99580bb71748-kube-api-access-9smpf\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.124948 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60abed7c-dc00-409b-9613-99580bb71748-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.124948 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60abed7c-dc00-409b-9613-99580bb71748-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.124948 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.124948 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.124948 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.124948 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60abed7c-dc00-409b-9613-99580bb71748-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.125174 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.124966 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226167 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226167 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smpf\" (UniqueName: \"kubernetes.io/projected/60abed7c-dc00-409b-9613-99580bb71748-kube-api-access-9smpf\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226167 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60abed7c-dc00-409b-9613-99580bb71748-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226167 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60abed7c-dc00-409b-9613-99580bb71748-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226456 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226456 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226456 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226456 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60abed7c-dc00-409b-9613-99580bb71748-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226456 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226737 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226737 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226704 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226836 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226786 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.226915 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.226894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.227043 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.227024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60abed7c-dc00-409b-9613-99580bb71748-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.228496 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.228466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60abed7c-dc00-409b-9613-99580bb71748-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.228701 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.228684 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60abed7c-dc00-409b-9613-99580bb71748-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.233744 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.233716 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60abed7c-dc00-409b-9613-99580bb71748-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.233912 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.233896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smpf\" (UniqueName: \"kubernetes.io/projected/60abed7c-dc00-409b-9613-99580bb71748-kube-api-access-9smpf\") pod \"maas-default-gateway-openshift-default-56654dcf7-pqbfw\" (UID: \"60abed7c-dc00-409b-9613-99580bb71748\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.285521 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.285495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:26.416401 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.416375 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw"] Jul 22 00:39:26.418205 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:39:26.418179 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60abed7c_dc00_409b_9613_99580bb71748.slice/crio-9494e261bf80211dd691c1910006af3d581860f149303feda59afbbe188b938b WatchSource:0}: Error finding container 9494e261bf80211dd691c1910006af3d581860f149303feda59afbbe188b938b: Status 404 returned error can't find the container with id 9494e261bf80211dd691c1910006af3d581860f149303feda59afbbe188b938b Jul 22 00:39:26.420215 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.420182 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:39:26.420322 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.420261 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:39:26.420322 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.420305 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:39:26.684442 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.684409 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" event={"ID":"60abed7c-dc00-409b-9613-99580bb71748","Type":"ContainerStarted","Data":"a26df14c3b3597c13ca94d9d77ddc78a9e32e017afa63705b8b1a03759e28a0e"} Jul 22 00:39:26.684623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.684465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" event={"ID":"60abed7c-dc00-409b-9613-99580bb71748","Type":"ContainerStarted","Data":"9494e261bf80211dd691c1910006af3d581860f149303feda59afbbe188b938b"} Jul 22 00:39:26.703276 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:26.703234 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" podStartSLOduration=1.703219971 podStartE2EDuration="1.703219971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:39:26.700746288 +0000 UTC m=+751.330978827" watchObservedRunningTime="2026-07-22 00:39:26.703219971 +0000 UTC m=+751.333452511" Jul 22 00:39:27.286290 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:27.286259 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:27.290710 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:27.290690 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:27.689269 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:27.689240 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:27.690371 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:27.690355 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:39:41.566264 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.566224 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:39:41.570692 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.570672 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:41.572997 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.572970 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mphx6\"" Jul 22 00:39:41.572997 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.572970 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 00:39:41.576987 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.576965 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:39:41.599118 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.599089 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:39:41.649601 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.649575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2df5edb-55ca-4a21-b04c-1b714c2564ce-config-file\") pod \"limitador-limitador-54d6d988d5-vv8vn\" (UID: \"e2df5edb-55ca-4a21-b04c-1b714c2564ce\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:41.649728 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.649611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjst2\" (UniqueName: \"kubernetes.io/projected/e2df5edb-55ca-4a21-b04c-1b714c2564ce-kube-api-access-jjst2\") pod \"limitador-limitador-54d6d988d5-vv8vn\" (UID: \"e2df5edb-55ca-4a21-b04c-1b714c2564ce\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:41.750016 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.749982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2df5edb-55ca-4a21-b04c-1b714c2564ce-config-file\") pod \"limitador-limitador-54d6d988d5-vv8vn\" (UID: \"e2df5edb-55ca-4a21-b04c-1b714c2564ce\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:41.750173 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.750028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjst2\" (UniqueName: \"kubernetes.io/projected/e2df5edb-55ca-4a21-b04c-1b714c2564ce-kube-api-access-jjst2\") pod \"limitador-limitador-54d6d988d5-vv8vn\" (UID: \"e2df5edb-55ca-4a21-b04c-1b714c2564ce\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:41.750873 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.750852 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2df5edb-55ca-4a21-b04c-1b714c2564ce-config-file\") pod \"limitador-limitador-54d6d988d5-vv8vn\" (UID: \"e2df5edb-55ca-4a21-b04c-1b714c2564ce\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:41.763378 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.763351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjst2\" (UniqueName: \"kubernetes.io/projected/e2df5edb-55ca-4a21-b04c-1b714c2564ce-kube-api-access-jjst2\") pod \"limitador-limitador-54d6d988d5-vv8vn\" (UID: \"e2df5edb-55ca-4a21-b04c-1b714c2564ce\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:41.882887 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:41.882823 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:42.006997 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:42.006966 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:39:42.009909 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:39:42.009879 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2df5edb_55ca_4a21_b04c_1b714c2564ce.slice/crio-54ed1ebe700782541e3ed88154cc2823f3977bb73b6273bd9d1133765cf0ca5e WatchSource:0}: Error finding container 54ed1ebe700782541e3ed88154cc2823f3977bb73b6273bd9d1133765cf0ca5e: Status 404 returned error can't find the container with id 54ed1ebe700782541e3ed88154cc2823f3977bb73b6273bd9d1133765cf0ca5e Jul 22 00:39:42.762409 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:42.762323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" event={"ID":"e2df5edb-55ca-4a21-b04c-1b714c2564ce","Type":"ContainerStarted","Data":"54ed1ebe700782541e3ed88154cc2823f3977bb73b6273bd9d1133765cf0ca5e"} Jul 22 00:39:44.773695 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:44.773591 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" event={"ID":"e2df5edb-55ca-4a21-b04c-1b714c2564ce","Type":"ContainerStarted","Data":"c131ea97b07342b0e5ef22549cbd4113701c4647a5b09ccfec7717eacb037bee"} Jul 22 00:39:44.774135 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:44.773711 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:39:44.790772 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:44.790723 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" podStartSLOduration=1.448604483 podStartE2EDuration="3.790709275s" totalImagesPullingTime="2.342104792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:39:44.789469695 +0000 UTC m=+769.419702236" watchObservedRunningTime="2026-07-22 00:39:44.790709275 +0000 UTC m=+769.420941814" Jul 22 00:39:55.779825 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:39:55.779791 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vv8vn" Jul 22 00:40:57.458779 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:40:57.458739 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:41:05.117393 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:05.117363 2562 generic.go:356] "Generic (PLEG): container finished" podID="60abed7c-dc00-409b-9613-99580bb71748" containerID="a26df14c3b3597c13ca94d9d77ddc78a9e32e017afa63705b8b1a03759e28a0e" exitCode=0 Jul 22 00:41:05.117773 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:05.117401 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" event={"ID":"60abed7c-dc00-409b-9613-99580bb71748","Type":"ContainerDied","Data":"a26df14c3b3597c13ca94d9d77ddc78a9e32e017afa63705b8b1a03759e28a0e"} Jul 22 00:41:05.117838 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:05.117785 2562 scope.go:117] "RemoveContainer" containerID="a26df14c3b3597c13ca94d9d77ddc78a9e32e017afa63705b8b1a03759e28a0e" Jul 22 00:41:05.118704 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:05.118677 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:41:05.118767 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:05.118740 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:41:05.118813 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:05.118773 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:41:06.128564 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:06.128533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" event={"ID":"60abed7c-dc00-409b-9613-99580bb71748","Type":"ContainerStarted","Data":"3cc03e6fb67c79cc418276c0ab6279a8bd7b0eac9902da8acc256e362b4393bc"} Jul 22 00:41:06.286437 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:06.286406 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:41:06.286604 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:06.286564 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pqbfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 22 00:41:06.286672 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:06.286615 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" podUID="60abed7c-dc00-409b-9613-99580bb71748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 22 00:41:07.286868 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:07.286828 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pqbfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 22 00:41:07.287248 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:07.286885 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" podUID="60abed7c-dc00-409b-9613-99580bb71748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 22 00:41:08.286155 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:08.286119 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pqbfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 22 00:41:08.286340 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:08.286181 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" podUID="60abed7c-dc00-409b-9613-99580bb71748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 22 00:41:09.286736 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:09.286699 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pqbfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 22 00:41:09.287131 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:09.286753 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" podUID="60abed7c-dc00-409b-9613-99580bb71748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 22 00:41:10.286685 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:10.286620 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pqbfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 22 00:41:10.286863 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:10.286712 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" podUID="60abed7c-dc00-409b-9613-99580bb71748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 22 00:41:11.289560 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:11.289531 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:41:11.289980 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:11.289905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:41:11.290426 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:11.290408 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pqbfw" Jul 22 00:41:31.016720 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:31.016685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:41:36.676093 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:36.676059 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:41:43.177863 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:43.177827 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:41:54.078436 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:54.078358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:41:55.843920 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:55.843892 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:41:55.844343 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:41:55.844039 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:42:05.370755 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:05.370723 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:42:22.599267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.599223 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv"] Jul 22 00:42:22.602585 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.602569 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.605765 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.605741 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jsfjh\"" Jul 22 00:42:22.605765 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.605767 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 00:42:22.605962 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.605768 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 00:42:22.605962 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.605799 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 00:42:22.611681 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.611633 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv"] Jul 22 00:42:22.722631 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.722601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.722832 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.722662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxsw\" (UniqueName: \"kubernetes.io/projected/5038e6ad-bd08-4ef8-84de-946d64735e72-kube-api-access-xkxsw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.722832 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.722728 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038e6ad-bd08-4ef8-84de-946d64735e72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.722832 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.722810 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.722935 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.722870 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.722935 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.722895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824098 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824175 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxsw\" (UniqueName: \"kubernetes.io/projected/5038e6ad-bd08-4ef8-84de-946d64735e72-kube-api-access-xkxsw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824267 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824238 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038e6ad-bd08-4ef8-84de-946d64735e72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824595 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824570 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824709 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.824709 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.824620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.826307 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.826281 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5038e6ad-bd08-4ef8-84de-946d64735e72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.826708 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.826689 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5038e6ad-bd08-4ef8-84de-946d64735e72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.831204 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.831182 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxsw\" (UniqueName: \"kubernetes.io/projected/5038e6ad-bd08-4ef8-84de-946d64735e72-kube-api-access-xkxsw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv\" (UID: \"5038e6ad-bd08-4ef8-84de-946d64735e72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:22.912919 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:22.912847 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:23.040897 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:23.040868 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv"] Jul 22 00:42:23.043405 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:42:23.043359 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5038e6ad_bd08_4ef8_84de_946d64735e72.slice/crio-93f3a3db85aa456dafc65612b51fcae188a1864fcdce5d86854bfe01b3df5f29 WatchSource:0}: Error finding container 93f3a3db85aa456dafc65612b51fcae188a1864fcdce5d86854bfe01b3df5f29: Status 404 returned error can't find the container with id 93f3a3db85aa456dafc65612b51fcae188a1864fcdce5d86854bfe01b3df5f29 Jul 22 00:42:23.482012 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:23.481971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" event={"ID":"5038e6ad-bd08-4ef8-84de-946d64735e72","Type":"ContainerStarted","Data":"93f3a3db85aa456dafc65612b51fcae188a1864fcdce5d86854bfe01b3df5f29"} Jul 22 00:42:24.488942 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:24.488847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" event={"ID":"5038e6ad-bd08-4ef8-84de-946d64735e72","Type":"ContainerStarted","Data":"07932585e83efc24f6cdc4aae7b6702bda51334673da208b4f62facdb2c00fa3"} Jul 22 00:42:24.506059 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:24.506007 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" podStartSLOduration=1.488747479 podStartE2EDuration="2.505985309s" totalImagesPullingTime="1.01723783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:42:24.503468842 +0000 UTC m=+929.133701379" watchObservedRunningTime="2026-07-22 00:42:24.505985309 +0000 UTC m=+929.136217850" Jul 22 00:42:25.274074 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:25.274040 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:42:32.913053 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:32.913018 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:32.913482 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:32.913080 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:32.925502 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:32.925479 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:42:33.539404 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:42:33.539376 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv" Jul 22 00:44:36.170393 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:44:36.170358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:44:45.271104 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:44:45.271070 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:44:55.972850 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:44:55.972765 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:46:39.677630 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:46:39.677546 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:46:55.870883 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:46:55.870853 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:46:55.871399 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:46:55.871382 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:47:19.569655 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:47:19.569618 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:47:29.077735 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:47:29.077699 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:47:37.773582 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:47:37.773552 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:47:47.068014 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:47:47.067978 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:47:57.975262 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:47:57.975225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:48:07.267899 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:48:07.267862 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:48:17.976817 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:48:17.976780 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:48:27.877037 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:48:27.877000 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:48:50.575782 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:48:50.575747 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:48:59.370498 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:48:59.370466 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:49:23.574935 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:49:23.574860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:49:34.370061 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:49:34.370029 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:49:49.183750 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:49:49.183712 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:49:59.080920 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:49:59.080888 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:50:08.579836 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:50:08.579800 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:50:19.272590 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:50:19.272555 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:50:28.379500 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:50:28.379464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:51:55.896097 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:51:55.896072 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:51:55.899309 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:51:55.899286 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:52:18.998951 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:52:18.998875 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:52:27.791169 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:52:27.791128 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:52:40.980826 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:52:40.980787 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:52:51.396284 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:52:51.396245 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:53:03.370076 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:53:03.370042 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:53:12.880126 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:53:12.880084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:53:23.478144 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:53:23.478112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:53:32.779434 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:53:32.779393 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:53:44.873178 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:53:44.873140 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:53:53.674849 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:53:53.674765 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:54:04.467606 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:04.467571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:54:25.676934 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:25.676892 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:54:34.972472 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:34.972440 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:54:38.620396 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.620358 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb"] Jul 22 00:54:38.624088 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.624066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.626536 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.626514 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7e056d38-kserve-self-signed-certs\"" Jul 22 00:54:38.637765 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.637744 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb"] Jul 22 00:54:38.736362 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.736332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728aadba-2987-480c-8b2c-dc972c55a7cc-tls-certs\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.736547 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.736407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-home\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.736547 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.736447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-tmp-dir\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.736547 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.736472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf5hx\" (UniqueName: \"kubernetes.io/projected/728aadba-2987-480c-8b2c-dc972c55a7cc-kube-api-access-cf5hx\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.736547 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.736530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-dshm\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.736747 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.736570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-model-cache\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837026 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.836998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-tmp-dir\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837209 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5hx\" (UniqueName: \"kubernetes.io/projected/728aadba-2987-480c-8b2c-dc972c55a7cc-kube-api-access-cf5hx\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837209 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-dshm\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837209 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-model-cache\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837209 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728aadba-2987-480c-8b2c-dc972c55a7cc-tls-certs\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837209 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-home\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837473 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-tmp-dir\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837532 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837492 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-model-cache\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.837595 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.837577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-home\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.839404 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.839374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-dshm\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.839589 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.839574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728aadba-2987-480c-8b2c-dc972c55a7cc-tls-certs\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.844181 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.844162 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5hx\" (UniqueName: \"kubernetes.io/projected/728aadba-2987-480c-8b2c-dc972c55a7cc-kube-api-access-cf5hx\") pod \"e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:38.933768 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:38.933692 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:39.068887 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:39.068861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb"] Jul 22 00:54:39.071095 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:54:39.071067 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod728aadba_2987_480c_8b2c_dc972c55a7cc.slice/crio-46b90ff405e59e00350003b95a5e9ded350f3cb496156fefa03b6b9a10c2262c WatchSource:0}: Error finding container 46b90ff405e59e00350003b95a5e9ded350f3cb496156fefa03b6b9a10c2262c: Status 404 returned error can't find the container with id 46b90ff405e59e00350003b95a5e9ded350f3cb496156fefa03b6b9a10c2262c Jul 22 00:54:39.774607 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:39.774568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" event={"ID":"728aadba-2987-480c-8b2c-dc972c55a7cc","Type":"ContainerStarted","Data":"c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4"} Jul 22 00:54:39.774607 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:39.774612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" event={"ID":"728aadba-2987-480c-8b2c-dc972c55a7cc","Type":"ContainerStarted","Data":"46b90ff405e59e00350003b95a5e9ded350f3cb496156fefa03b6b9a10c2262c"} Jul 22 00:54:39.792060 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:39.792013 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" podStartSLOduration=1.791998429 podStartE2EDuration="1.791998429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:54:39.7900961 +0000 UTC m=+1664.420328640" watchObservedRunningTime="2026-07-22 00:54:39.791998429 +0000 UTC m=+1664.422230968" Jul 22 00:54:48.934365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:48.934280 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:48.934365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:48.934326 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:48.946004 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:48.945985 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:54:48.975999 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:48.975963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:54:49.829950 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:54:49.829921 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:55:05.984946 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:05.984914 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb"] Jul 22 00:55:10.270687 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:10.270636 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:55:21.004855 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.004751 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" podUID="728aadba-2987-480c-8b2c-dc972c55a7cc" containerName="main" containerID="cri-o://c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4" gracePeriod=45 Jul 22 00:55:21.250017 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.249986 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:55:21.395737 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.395709 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-tmp-dir\") pod \"728aadba-2987-480c-8b2c-dc972c55a7cc\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " Jul 22 00:55:21.395911 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.395741 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-home\") pod \"728aadba-2987-480c-8b2c-dc972c55a7cc\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " Jul 22 00:55:21.395911 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.395779 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728aadba-2987-480c-8b2c-dc972c55a7cc-tls-certs\") pod \"728aadba-2987-480c-8b2c-dc972c55a7cc\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " Jul 22 00:55:21.395911 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.395837 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-dshm\") pod \"728aadba-2987-480c-8b2c-dc972c55a7cc\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " Jul 22 00:55:21.395911 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.395873 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf5hx\" (UniqueName: \"kubernetes.io/projected/728aadba-2987-480c-8b2c-dc972c55a7cc-kube-api-access-cf5hx\") pod \"728aadba-2987-480c-8b2c-dc972c55a7cc\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " Jul 22 00:55:21.396123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.395935 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-model-cache\") pod \"728aadba-2987-480c-8b2c-dc972c55a7cc\" (UID: \"728aadba-2987-480c-8b2c-dc972c55a7cc\") " Jul 22 00:55:21.396123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.395968 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "728aadba-2987-480c-8b2c-dc972c55a7cc" (UID: "728aadba-2987-480c-8b2c-dc972c55a7cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:21.396123 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.396015 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-home" (OuterVolumeSpecName: "home") pod "728aadba-2987-480c-8b2c-dc972c55a7cc" (UID: "728aadba-2987-480c-8b2c-dc972c55a7cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:21.396259 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.396201 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-tmp-dir\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:55:21.396259 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.396216 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-home\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:55:21.396259 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.396221 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "728aadba-2987-480c-8b2c-dc972c55a7cc" (UID: "728aadba-2987-480c-8b2c-dc972c55a7cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:21.398129 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.398100 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-dshm" (OuterVolumeSpecName: "dshm") pod "728aadba-2987-480c-8b2c-dc972c55a7cc" (UID: "728aadba-2987-480c-8b2c-dc972c55a7cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:55:21.398234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.398146 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728aadba-2987-480c-8b2c-dc972c55a7cc-kube-api-access-cf5hx" (OuterVolumeSpecName: "kube-api-access-cf5hx") pod "728aadba-2987-480c-8b2c-dc972c55a7cc" (UID: "728aadba-2987-480c-8b2c-dc972c55a7cc"). InnerVolumeSpecName "kube-api-access-cf5hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:55:21.398234 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.398195 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728aadba-2987-480c-8b2c-dc972c55a7cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "728aadba-2987-480c-8b2c-dc972c55a7cc" (UID: "728aadba-2987-480c-8b2c-dc972c55a7cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:55:21.497713 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.497684 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728aadba-2987-480c-8b2c-dc972c55a7cc-tls-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:55:21.497713 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.497709 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-dshm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:55:21.497713 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.497718 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf5hx\" (UniqueName: \"kubernetes.io/projected/728aadba-2987-480c-8b2c-dc972c55a7cc-kube-api-access-cf5hx\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:55:21.497917 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.497726 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728aadba-2987-480c-8b2c-dc972c55a7cc-model-cache\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:55:21.961613 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.961578 2562 generic.go:356] "Generic (PLEG): container finished" podID="728aadba-2987-480c-8b2c-dc972c55a7cc" containerID="c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4" exitCode=0 Jul 22 00:55:21.961768 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.961620 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" Jul 22 00:55:21.961768 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.961666 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" event={"ID":"728aadba-2987-480c-8b2c-dc972c55a7cc","Type":"ContainerDied","Data":"c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4"} Jul 22 00:55:21.961768 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.961708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb" event={"ID":"728aadba-2987-480c-8b2c-dc972c55a7cc","Type":"ContainerDied","Data":"46b90ff405e59e00350003b95a5e9ded350f3cb496156fefa03b6b9a10c2262c"} Jul 22 00:55:21.961768 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.961725 2562 scope.go:117] "RemoveContainer" containerID="c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4" Jul 22 00:55:21.969884 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.969868 2562 scope.go:117] "RemoveContainer" containerID="c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4" Jul 22 00:55:21.970140 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:55:21.970121 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4\": container with ID starting with c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4 not found: ID does not exist" containerID="c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4" Jul 22 00:55:21.970210 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.970147 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4"} err="failed to get container status \"c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4\": rpc error: code = NotFound desc = could not find container \"c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4\": container with ID starting with c1fe9099fffcdd70346497824c482dd6ea5e8ccf2768ff1990a45e919a516fa4 not found: ID does not exist" Jul 22 00:55:21.976655 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.976620 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb"] Jul 22 00:55:21.978613 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:21.978594 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7e056d38-kserve-58cdcbcd87-dfvkb"] Jul 22 00:55:23.378247 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:23.378211 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:55:23.905853 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:23.905822 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="728aadba-2987-480c-8b2c-dc972c55a7cc" path="/var/lib/kubelet/pods/728aadba-2987-480c-8b2c-dc972c55a7cc/volumes" Jul 22 00:55:31.274037 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:31.273998 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:55:46.780049 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:46.780014 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:55:56.369955 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:55:56.369908 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:56:05.672308 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:05.672271 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:56:13.375496 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:13.375463 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:56:24.573628 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:24.573593 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:56:33.874364 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:33.874333 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:56:47.681398 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:47.681310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:56:55.923788 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:55.923690 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:56:55.927193 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:55.927174 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 00:56:56.881670 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:56:56.881622 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:57:06.877910 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:57:06.877874 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:57:16.473623 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:57:16.473588 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:57:28.969190 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:57:28.969150 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:57:37.874252 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:57:37.874214 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:57:50.079948 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:57:50.079909 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:58:00.786168 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:00.786128 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:58:11.277699 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:11.277667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:58:24.697353 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:24.697262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:58:34.470276 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:34.470232 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:58:44.474495 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:44.474461 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:58:44.984868 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:44.984831 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp"] Jul 22 00:58:44.985214 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:44.985201 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="728aadba-2987-480c-8b2c-dc972c55a7cc" containerName="main" Jul 22 00:58:44.985257 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:44.985216 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="728aadba-2987-480c-8b2c-dc972c55a7cc" containerName="main" Jul 22 00:58:44.985290 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:44.985275 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="728aadba-2987-480c-8b2c-dc972c55a7cc" containerName="main" Jul 22 00:58:44.987874 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:44.987858 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:44.990090 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:44.990075 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-827fc75f-openshift-default-dockercfg-fdpbv\"" Jul 22 00:58:45.008473 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.008446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp"] Jul 22 00:58:45.162306 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-certs\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-credential-socket\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wct8t\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-kube-api-access-wct8t\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-data\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162430 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-socket\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162720 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-podinfo\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162720 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istiod-ca-cert\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162720 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-envoy\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.162720 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.162659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-token\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263280 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-certs\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263280 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-credential-socket\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263280 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wct8t\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-kube-api-access-wct8t\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263280 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-data\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263571 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-socket\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263571 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-podinfo\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263571 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istiod-ca-cert\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263571 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-envoy\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263571 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-token\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263808 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-certs\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263808 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-socket\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263914 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263899 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-data\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.263977 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.263947 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-credential-socket\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.264176 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.264153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istiod-ca-cert\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.265768 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.265746 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-envoy\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.265868 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.265837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-podinfo\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.271063 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.271044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-token\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.271137 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.271086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wct8t\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-kube-api-access-wct8t\") pod \"e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.298380 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.298353 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:45.433708 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.433680 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp"] Jul 22 00:58:45.435617 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:58:45.435592 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc5a4e5_33bc_4e8b_8a0e_b3468cdccbbf.slice/crio-70019c0acda976b9c3cfe8bc342bc1ed5f285bdc37346f6d513f36fbe20b9606 WatchSource:0}: Error finding container 70019c0acda976b9c3cfe8bc342bc1ed5f285bdc37346f6d513f36fbe20b9606: Status 404 returned error can't find the container with id 70019c0acda976b9c3cfe8bc342bc1ed5f285bdc37346f6d513f36fbe20b9606 Jul 22 00:58:45.437741 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.437709 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:58:45.437808 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.437775 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:58:45.437808 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.437800 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:58:45.873682 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.873623 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" event={"ID":"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf","Type":"ContainerStarted","Data":"f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38"} Jul 22 00:58:45.874040 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.873701 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" event={"ID":"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf","Type":"ContainerStarted","Data":"70019c0acda976b9c3cfe8bc342bc1ed5f285bdc37346f6d513f36fbe20b9606"} Jul 22 00:58:45.893023 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:45.892980 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" podStartSLOduration=1.892964894 podStartE2EDuration="1.892964894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:58:45.892023987 +0000 UTC m=+1910.522256528" watchObservedRunningTime="2026-07-22 00:58:45.892964894 +0000 UTC m=+1910.523197433" Jul 22 00:58:46.298533 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:46.298499 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:46.304210 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:46.304185 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:46.877742 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:46.877715 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:46.878726 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:46.878709 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:58:54.472578 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:58:54.472541 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:59:02.138144 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.138112 2562 patch_prober.go:28] interesting pod/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": read tcp 10.134.0.2:39240->10.134.0.31:15021: read: connection reset by peer" start-of-body= Jul 22 00:59:02.138453 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.138199 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": read tcp 10.134.0.2:39240->10.134.0.31:15021: read: connection reset by peer" Jul 22 00:59:02.946619 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.946585 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerID="f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38" exitCode=0 Jul 22 00:59:02.946804 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.946671 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" event={"ID":"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf","Type":"ContainerDied","Data":"f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38"} Jul 22 00:59:02.947038 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.947014 2562 scope.go:117] "RemoveContainer" containerID="f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38" Jul 22 00:59:02.947975 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.947945 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:02.948039 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.948028 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:02.948088 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:02.948072 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:03.955664 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:03.955605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" event={"ID":"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf","Type":"ContainerStarted","Data":"9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8"} Jul 22 00:59:04.299157 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:04.299123 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:59:04.299364 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:04.299342 2562 patch_prober.go:28] interesting pod/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 00:59:04.299410 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:04.299393 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 00:59:05.299662 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:05.299606 2562 patch_prober.go:28] interesting pod/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 00:59:05.300070 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:05.299684 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 00:59:06.298766 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:06.298728 2562 patch_prober.go:28] interesting pod/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 00:59:06.299029 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:06.298791 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 00:59:07.299501 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:07.299465 2562 patch_prober.go:28] interesting pod/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 22 00:59:07.299912 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:07.299519 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 22 00:59:08.302435 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:08.302406 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:59:08.302863 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:08.302673 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:59:08.303383 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:08.303366 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:59:26.169143 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:26.169112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:59:27.986653 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:27.986612 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp"] Jul 22 00:59:27.987050 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:27.986836 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" containerID="cri-o://9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8" gracePeriod=30 Jul 22 00:59:28.964462 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:28.964424 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh"] Jul 22 00:59:28.967204 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:28.967184 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:28.969332 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:28.969308 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f98ed7b9-gw-openshift-default-dockercfg-kmd4f\"" Jul 22 00:59:28.977541 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:28.977518 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh"] Jul 22 00:59:29.061566 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5b5220-e1c6-467d-af0e-8357d5470426-istiod-ca-cert\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-data\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkfw\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-kube-api-access-9gkfw\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-envoy\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-token\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-credential-socket\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-certs\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-podinfo\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.061978 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.061929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-socket\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163075 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163039 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-credential-socket\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163075 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-certs\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163302 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-podinfo\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163302 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-socket\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163302 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5b5220-e1c6-467d-af0e-8357d5470426-istiod-ca-cert\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163302 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163194 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-data\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163302 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkfw\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-kube-api-access-9gkfw\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163597 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163508 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-credential-socket\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163597 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163521 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-certs\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163597 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-envoy\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163597 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-token\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163810 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-data\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163848 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-socket\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.163941 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.163922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5b5220-e1c6-467d-af0e-8357d5470426-istiod-ca-cert\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.165572 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.165552 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-envoy\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.165856 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.165837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-podinfo\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.171069 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.171045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-token\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.171204 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.171188 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkfw\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-kube-api-access-9gkfw\") pod \"e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.277231 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.277164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:29.414887 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.410349 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh"] Jul 22 00:59:29.417219 ip-10-0-131-63 kubenswrapper[2562]: W0722 00:59:29.417169 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5b5220_e1c6_467d_af0e_8357d5470426.slice/crio-840778556c8a2b9fc3051c5f5de9fea8c018ceb2df17d0d08812cf3c123809da WatchSource:0}: Error finding container 840778556c8a2b9fc3051c5f5de9fea8c018ceb2df17d0d08812cf3c123809da: Status 404 returned error can't find the container with id 840778556c8a2b9fc3051c5f5de9fea8c018ceb2df17d0d08812cf3c123809da Jul 22 00:59:29.419693 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.419628 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:29.419789 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.419728 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:29.419789 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:29.419759 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:30.078553 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:30.078522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" event={"ID":"ff5b5220-e1c6-467d-af0e-8357d5470426","Type":"ContainerStarted","Data":"93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f"} Jul 22 00:59:30.078946 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:30.078579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" event={"ID":"ff5b5220-e1c6-467d-af0e-8357d5470426","Type":"ContainerStarted","Data":"840778556c8a2b9fc3051c5f5de9fea8c018ceb2df17d0d08812cf3c123809da"} Jul 22 00:59:30.098085 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:30.098016 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podStartSLOduration=2.097998156 podStartE2EDuration="2.097998156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:59:30.097579087 +0000 UTC m=+1954.727811639" watchObservedRunningTime="2026-07-22 00:59:30.097998156 +0000 UTC m=+1954.728230698" Jul 22 00:59:30.277691 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:30.277636 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:30.282063 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:30.282040 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:31.083577 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:31.083548 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:31.084290 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:31.084274 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:33.243122 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.243096 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:59:33.400548 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400519 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-credential-socket\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.400751 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400571 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-podinfo\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.400751 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400587 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-token\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.400751 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400634 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istiod-ca-cert\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.400751 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400687 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-envoy\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.400751 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400717 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wct8t\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-kube-api-access-wct8t\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.401005 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400748 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-data\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.401005 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400782 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-socket\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.401005 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400811 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-certs\") pod \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\" (UID: \"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf\") " Jul 22 00:59:33.401005 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.400844 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:59:33.401204 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.401068 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:59:33.401204 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.401148 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.401204 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.401168 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.401204 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.401172 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:59:33.401413 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.401251 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:59:33.401549 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.401527 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:59:33.403082 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.403053 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:59:33.403363 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.403064 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-kube-api-access-wct8t" (OuterVolumeSpecName: "kube-api-access-wct8t") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "kube-api-access-wct8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:59:33.403363 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.403245 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:59:33.403363 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.403264 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" (UID: "bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 00:59:33.502058 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.502028 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.502058 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.502053 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.502058 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.502063 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.502271 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.502071 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.502271 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.502078 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.502271 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.502085 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:33.502271 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:33.502093 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wct8t\" (UniqueName: \"kubernetes.io/projected/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf-kube-api-access-wct8t\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 00:59:34.099309 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.099280 2562 generic.go:356] "Generic (PLEG): container finished" podID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerID="9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8" exitCode=0 Jul 22 00:59:34.099491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.099335 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" Jul 22 00:59:34.099491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.099365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" event={"ID":"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf","Type":"ContainerDied","Data":"9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8"} Jul 22 00:59:34.099491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.099420 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp" event={"ID":"bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf","Type":"ContainerDied","Data":"70019c0acda976b9c3cfe8bc342bc1ed5f285bdc37346f6d513f36fbe20b9606"} Jul 22 00:59:34.099491 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.099442 2562 scope.go:117] "RemoveContainer" containerID="9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8" Jul 22 00:59:34.107380 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.107205 2562 scope.go:117] "RemoveContainer" containerID="f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38" Jul 22 00:59:34.114516 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.114443 2562 scope.go:117] "RemoveContainer" containerID="9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8" Jul 22 00:59:34.114758 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:59:34.114729 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8\": container with ID starting with 9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8 not found: ID does not exist" containerID="9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8" Jul 22 00:59:34.114841 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.114769 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8"} err="failed to get container status \"9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8\": rpc error: code = NotFound desc = could not find container \"9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8\": container with ID starting with 9dd669347ba9680533a524bcdd5c5ee6d3b4b8c434a6831ef3aa927896343fb8 not found: ID does not exist" Jul 22 00:59:34.114841 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.114790 2562 scope.go:117] "RemoveContainer" containerID="f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38" Jul 22 00:59:34.115114 ip-10-0-131-63 kubenswrapper[2562]: E0722 00:59:34.115094 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38\": container with ID starting with f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38 not found: ID does not exist" containerID="f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38" Jul 22 00:59:34.115175 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.115120 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38"} err="failed to get container status \"f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38\": rpc error: code = NotFound desc = could not find container \"f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38\": container with ID starting with f2dceb72d148c0578328344be6df10810d79811ee111ab80e40ed6d664b1cc38 not found: ID does not exist" Jul 22 00:59:34.116778 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.116760 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp"] Jul 22 00:59:34.120038 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:34.120019 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-827fc75f-openshift-default-7956755ff-kfpqp"] Jul 22 00:59:35.906368 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:35.906335 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" path="/var/lib/kubelet/pods/bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf/volumes" Jul 22 00:59:38.370014 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:38.369975 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 00:59:46.085079 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.085042 2562 patch_prober.go:28] interesting pod/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 22 00:59:46.085549 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.085100 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 22 00:59:46.158365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.158291 2562 generic.go:356] "Generic (PLEG): container finished" podID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerID="93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f" exitCode=0 Jul 22 00:59:46.158365 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.158350 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" event={"ID":"ff5b5220-e1c6-467d-af0e-8357d5470426","Type":"ContainerDied","Data":"93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f"} Jul 22 00:59:46.158721 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.158706 2562 scope.go:117] "RemoveContainer" containerID="93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f" Jul 22 00:59:46.159638 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.159607 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:46.159777 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.159695 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:46.159777 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:46.159730 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 00:59:47.164694 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:47.164661 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" event={"ID":"ff5b5220-e1c6-467d-af0e-8357d5470426","Type":"ContainerStarted","Data":"63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b"} Jul 22 00:59:47.278025 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:47.277993 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:47.278214 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:47.278190 2562 patch_prober.go:28] interesting pod/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 22 00:59:47.278283 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:47.278244 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 22 00:59:48.278452 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:48.278417 2562 patch_prober.go:28] interesting pod/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 22 00:59:48.278849 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:48.278468 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 22 00:59:49.278616 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:49.278578 2562 patch_prober.go:28] interesting pod/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 22 00:59:49.279040 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:49.278631 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 22 00:59:50.278548 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:50.278514 2562 patch_prober.go:28] interesting pod/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 22 00:59:50.278743 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:50.278575 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 22 00:59:51.277766 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:51.277731 2562 patch_prober.go:28] interesting pod/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 22 00:59:51.277940 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:51.277790 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 22 00:59:52.281174 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:52.281141 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:52.281580 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:52.281395 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 00:59:52.282100 ip-10-0-131-63 kubenswrapper[2562]: I0722 00:59:52.282086 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 01:00:09.776297 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:09.776265 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:00:11.768903 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:11.768871 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh"] Jul 22 01:00:11.769316 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:11.769076 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" containerID="cri-o://63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b" gracePeriod=30 Jul 22 01:00:12.582954 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.582925 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx"] Jul 22 01:00:12.583358 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.583341 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" Jul 22 01:00:12.583442 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.583360 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" Jul 22 01:00:12.583492 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.583462 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" Jul 22 01:00:12.586467 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.586448 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.588558 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.588540 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b80956c0-openshift-default-dockercfg-qz9m5\"" Jul 22 01:00:12.598638 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.598617 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx"] Jul 22 01:00:12.754540 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-envoy\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.754791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754580 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-certs\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.754791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af54c397-300d-4945-9440-e71003e21f39-istio-podinfo\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.754791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-istio-token\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.754791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af54c397-300d-4945-9440-e71003e21f39-istiod-ca-cert\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.754791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-credential-socket\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.754791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754771 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-data\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.754791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtcsh\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-kube-api-access-jtcsh\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.755164 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.754852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-socket\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.855550 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-certs\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.855550 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af54c397-300d-4945-9440-e71003e21f39-istio-podinfo\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-istio-token\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af54c397-300d-4945-9440-e71003e21f39-istiod-ca-cert\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-credential-socket\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-data\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtcsh\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-kube-api-access-jtcsh\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-socket\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855898 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-envoy\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.855920 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-certs\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856437 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.856128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-credential-socket\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856437 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.856265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-socket\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856437 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.856267 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af54c397-300d-4945-9440-e71003e21f39-istiod-ca-cert\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.856437 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.856315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-data\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.858018 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.857997 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af54c397-300d-4945-9440-e71003e21f39-istio-podinfo\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.858109 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.858028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-envoy\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.863062 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.863041 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-istio-token\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.863193 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.863177 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtcsh\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-kube-api-access-jtcsh\") pod \"e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:12.896244 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:12.896223 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:13.025181 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.025159 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx"] Jul 22 01:00:13.027479 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:00:13.027451 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf54c397_300d_4945_9440_e71003e21f39.slice/crio-c1dce2baf32704a3579d442aa4d9ee0cc0a2c9ce98202887a51bc25d35687379 WatchSource:0}: Error finding container c1dce2baf32704a3579d442aa4d9ee0cc0a2c9ce98202887a51bc25d35687379: Status 404 returned error can't find the container with id c1dce2baf32704a3579d442aa4d9ee0cc0a2c9ce98202887a51bc25d35687379 Jul 22 01:00:13.029462 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.029429 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:13.029552 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.029492 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:13.029552 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.029540 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:13.284231 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.284133 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" event={"ID":"af54c397-300d-4945-9440-e71003e21f39","Type":"ContainerStarted","Data":"e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c"} Jul 22 01:00:13.284231 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.284185 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" event={"ID":"af54c397-300d-4945-9440-e71003e21f39","Type":"ContainerStarted","Data":"c1dce2baf32704a3579d442aa4d9ee0cc0a2c9ce98202887a51bc25d35687379"} Jul 22 01:00:13.319881 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.319832 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" podStartSLOduration=1.31981514 podStartE2EDuration="1.31981514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:00:13.317710105 +0000 UTC m=+1997.947942644" watchObservedRunningTime="2026-07-22 01:00:13.31981514 +0000 UTC m=+1997.950047683" Jul 22 01:00:13.896868 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:13.896838 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:14.901781 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:14.901753 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:15.294222 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:15.294194 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:15.295147 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:15.295129 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:17.014451 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.014428 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 01:00:17.197686 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-envoy\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197686 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197700 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-token\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197729 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gkfw\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-kube-api-access-9gkfw\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197752 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-socket\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197778 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-credential-socket\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197799 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5b5220-e1c6-467d-af0e-8357d5470426-istiod-ca-cert\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-data\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197866 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-podinfo\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.197941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.197902 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-certs\") pod \"ff5b5220-e1c6-467d-af0e-8357d5470426\" (UID: \"ff5b5220-e1c6-467d-af0e-8357d5470426\") " Jul 22 01:00:17.198318 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.198156 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:00:17.198318 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.198151 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:00:17.198318 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.198225 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.198470 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.198375 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:00:17.198470 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.198427 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5b5220-e1c6-467d-af0e-8357d5470426-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:00:17.198819 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.198799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:00:17.199974 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.199957 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-kube-api-access-9gkfw" (OuterVolumeSpecName: "kube-api-access-9gkfw") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "kube-api-access-9gkfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:00:17.200063 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.200029 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:00:17.200249 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.200227 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:00:17.200343 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.200228 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff5b5220-e1c6-467d-af0e-8357d5470426" (UID: "ff5b5220-e1c6-467d-af0e-8357d5470426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:00:17.299504 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299476 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.299504 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299503 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.299504 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299517 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gkfw\" (UniqueName: \"kubernetes.io/projected/ff5b5220-e1c6-467d-af0e-8357d5470426-kube-api-access-9gkfw\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.299811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299530 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.299811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299544 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5b5220-e1c6-467d-af0e-8357d5470426-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.299811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299554 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.299811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299565 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5b5220-e1c6-467d-af0e-8357d5470426-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.299811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.299575 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5b5220-e1c6-467d-af0e-8357d5470426-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:00:17.311054 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.311021 2562 generic.go:356] "Generic (PLEG): container finished" podID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerID="63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b" exitCode=0 Jul 22 01:00:17.311201 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.311093 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" Jul 22 01:00:17.311201 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.311097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" event={"ID":"ff5b5220-e1c6-467d-af0e-8357d5470426","Type":"ContainerDied","Data":"63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b"} Jul 22 01:00:17.311201 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.311165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh" event={"ID":"ff5b5220-e1c6-467d-af0e-8357d5470426","Type":"ContainerDied","Data":"840778556c8a2b9fc3051c5f5de9fea8c018ceb2df17d0d08812cf3c123809da"} Jul 22 01:00:17.311201 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.311186 2562 scope.go:117] "RemoveContainer" containerID="63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b" Jul 22 01:00:17.324390 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.324374 2562 scope.go:117] "RemoveContainer" containerID="93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f" Jul 22 01:00:17.332486 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.332470 2562 scope.go:117] "RemoveContainer" containerID="63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b" Jul 22 01:00:17.332765 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:00:17.332739 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b\": container with ID starting with 63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b not found: ID does not exist" containerID="63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b" Jul 22 01:00:17.332833 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.332777 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b"} err="failed to get container status \"63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b\": rpc error: code = NotFound desc = could not find container \"63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b\": container with ID starting with 63e8d8d59849090ab76d901bfb5d11ddceee932411a238ece69459961a34d88b not found: ID does not exist" Jul 22 01:00:17.332833 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.332799 2562 scope.go:117] "RemoveContainer" containerID="93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f" Jul 22 01:00:17.333040 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:00:17.333021 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f\": container with ID starting with 93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f not found: ID does not exist" containerID="93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f" Jul 22 01:00:17.333101 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.333048 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f"} err="failed to get container status \"93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f\": rpc error: code = NotFound desc = could not find container \"93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f\": container with ID starting with 93855b3f17a72aabd7fb198e1d9f8519598a0c88543405f1a3a9b93d1b573e9f not found: ID does not exist" Jul 22 01:00:17.335675 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.335653 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh"] Jul 22 01:00:17.341506 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.341488 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f98ed7b9-gw-openshift-default-ccfffc8f9-6x7bh"] Jul 22 01:00:17.905583 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:17.905549 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" path="/var/lib/kubelet/pods/ff5b5220-e1c6-467d-af0e-8357d5470426/volumes" Jul 22 01:00:21.779334 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:21.779299 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:00:31.335815 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:31.335778 2562 patch_prober.go:28] interesting pod/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 01:00:31.336215 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:31.335843 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 01:00:32.393580 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:32.393544 2562 generic.go:356] "Generic (PLEG): container finished" podID="af54c397-300d-4945-9440-e71003e21f39" containerID="e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c" exitCode=0 Jul 22 01:00:32.394039 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:32.393614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" event={"ID":"af54c397-300d-4945-9440-e71003e21f39","Type":"ContainerDied","Data":"e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c"} Jul 22 01:00:32.394039 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:32.393986 2562 scope.go:117] "RemoveContainer" containerID="e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c" Jul 22 01:00:32.394907 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:32.394873 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:32.395008 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:32.394942 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:32.395008 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:32.394970 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:33.399836 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:33.399800 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" event={"ID":"af54c397-300d-4945-9440-e71003e21f39","Type":"ContainerStarted","Data":"6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9"} Jul 22 01:00:33.896569 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:33.896537 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:33.896878 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:33.896859 2562 patch_prober.go:28] interesting pod/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 01:00:33.896926 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:33.896909 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 01:00:34.897467 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:34.897433 2562 patch_prober.go:28] interesting pod/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 01:00:34.897900 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:34.897485 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 01:00:35.896629 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:35.896582 2562 patch_prober.go:28] interesting pod/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 01:00:35.896805 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:35.896669 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 01:00:36.897191 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:36.897153 2562 patch_prober.go:28] interesting pod/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 22 01:00:36.897588 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:36.897208 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 22 01:00:37.905094 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:37.905067 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:37.905483 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:37.905428 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:37.906176 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:37.906156 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:00:52.881600 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:52.881562 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:00:56.729919 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:56.729881 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx"] Jul 22 01:00:56.730321 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:56.730209 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" containerID="cri-o://6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9" gracePeriod=30 Jul 22 01:00:57.519517 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519486 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz"] Jul 22 01:00:57.519892 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519879 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" Jul 22 01:00:57.519941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519895 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" Jul 22 01:00:57.519941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519915 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" Jul 22 01:00:57.519941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519920 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" Jul 22 01:00:57.519941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519928 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" Jul 22 01:00:57.519941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519934 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" Jul 22 01:00:57.520094 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519982 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" Jul 22 01:00:57.520094 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519990 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5b5220-e1c6-467d-af0e-8357d5470426" containerName="istio-proxy" Jul 22 01:00:57.520094 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.519998 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc5a4e5-33bc-4e8b-8a0e-b3468cdccbbf" containerName="istio-proxy" Jul 22 01:00:57.523760 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.523732 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.526053 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.526030 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5186cc87-openshift-default-dockercfg-w24dj\"" Jul 22 01:00:57.533488 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.533437 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz"] Jul 22 01:00:57.647261 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-certs\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647440 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-envoy\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647440 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-socket\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647440 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istiod-ca-cert\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647607 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gngc\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-kube-api-access-7gngc\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647607 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-data\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647607 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-podinfo\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647782 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647656 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-credential-socket\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.647782 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.647688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-token\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.748950 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.748916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-certs\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.748950 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.748956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-envoy\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.748982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-socket\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istiod-ca-cert\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gngc\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-kube-api-access-7gngc\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-data\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-podinfo\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-credential-socket\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-token\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-socket\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749964 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-certs\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749964 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-data\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.749964 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.749943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-credential-socket\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.750088 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.750036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istiod-ca-cert\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.751746 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.751723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-envoy\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.751852 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.751772 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-podinfo\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.756365 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.756343 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-token\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.756617 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.756600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gngc\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-kube-api-access-7gngc\") pod \"e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.834172 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.834143 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:57.967805 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.967775 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz"] Jul 22 01:00:57.969947 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:00:57.969922 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c625cf3_7fe4_4846_bb7f_8ffed3f2e402.slice/crio-18dc17583e95723cd33de5804f87b2b2bd0f8322d56e8585d01280693519fec3 WatchSource:0}: Error finding container 18dc17583e95723cd33de5804f87b2b2bd0f8322d56e8585d01280693519fec3: Status 404 returned error can't find the container with id 18dc17583e95723cd33de5804f87b2b2bd0f8322d56e8585d01280693519fec3 Jul 22 01:00:57.972199 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.972164 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:57.972291 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.972228 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:57.972291 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:57.972275 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:00:58.518238 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:58.518200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" event={"ID":"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402","Type":"ContainerStarted","Data":"3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b"} Jul 22 01:00:58.518419 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:58.518268 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" event={"ID":"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402","Type":"ContainerStarted","Data":"18dc17583e95723cd33de5804f87b2b2bd0f8322d56e8585d01280693519fec3"} Jul 22 01:00:58.540657 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:58.540600 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podStartSLOduration=1.540582983 podStartE2EDuration="1.540582983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:00:58.539424882 +0000 UTC m=+2043.169657426" watchObservedRunningTime="2026-07-22 01:00:58.540582983 +0000 UTC m=+2043.170815524" Jul 22 01:00:58.834289 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:58.834260 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:00:59.839664 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:00:59.839625 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:00.527936 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:00.527895 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:00.528841 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:00.528824 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:01.971235 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:01.971210 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:01:02.095348 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095310 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-data\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095541 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095372 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-socket\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095541 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095414 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtcsh\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-kube-api-access-jtcsh\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095541 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095434 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-credential-socket\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095541 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095467 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-certs\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095541 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095487 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-envoy\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095541 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095519 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af54c397-300d-4945-9440-e71003e21f39-istiod-ca-cert\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095902 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095557 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-istio-token\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095902 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af54c397-300d-4945-9440-e71003e21f39-istio-podinfo\") pod \"af54c397-300d-4945-9440-e71003e21f39\" (UID: \"af54c397-300d-4945-9440-e71003e21f39\") " Jul 22 01:01:02.095902 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095882 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:02.096052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.095893 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:02.096052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.096025 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af54c397-300d-4945-9440-e71003e21f39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:01:02.096163 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.096129 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:02.096286 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.096263 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-data" (OuterVolumeSpecName: "istio-data") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:02.097901 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.097871 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-istio-token" (OuterVolumeSpecName: "istio-token") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:01:02.097995 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.097942 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af54c397-300d-4945-9440-e71003e21f39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:01:02.098108 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.098089 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:02.098174 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.098158 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-kube-api-access-jtcsh" (OuterVolumeSpecName: "kube-api-access-jtcsh") pod "af54c397-300d-4945-9440-e71003e21f39" (UID: "af54c397-300d-4945-9440-e71003e21f39"). InnerVolumeSpecName "kube-api-access-jtcsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:01:02.196761 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196726 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtcsh\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-kube-api-access-jtcsh\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196761 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196755 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196761 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196765 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196775 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196784 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af54c397-300d-4945-9440-e71003e21f39-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196792 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af54c397-300d-4945-9440-e71003e21f39-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196799 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af54c397-300d-4945-9440-e71003e21f39-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196807 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.196985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.196814 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af54c397-300d-4945-9440-e71003e21f39-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:02.538218 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.538129 2562 generic.go:356] "Generic (PLEG): container finished" podID="af54c397-300d-4945-9440-e71003e21f39" containerID="6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9" exitCode=0 Jul 22 01:01:02.538218 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.538174 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" Jul 22 01:01:02.538401 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.538217 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" event={"ID":"af54c397-300d-4945-9440-e71003e21f39","Type":"ContainerDied","Data":"6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9"} Jul 22 01:01:02.538401 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.538270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx" event={"ID":"af54c397-300d-4945-9440-e71003e21f39","Type":"ContainerDied","Data":"c1dce2baf32704a3579d442aa4d9ee0cc0a2c9ce98202887a51bc25d35687379"} Jul 22 01:01:02.538401 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.538293 2562 scope.go:117] "RemoveContainer" containerID="6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9" Jul 22 01:01:02.547795 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.547777 2562 scope.go:117] "RemoveContainer" containerID="e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c" Jul 22 01:01:02.555099 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.555083 2562 scope.go:117] "RemoveContainer" containerID="6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9" Jul 22 01:01:02.555360 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:01:02.555341 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9\": container with ID starting with 6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9 not found: ID does not exist" containerID="6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9" Jul 22 01:01:02.555431 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.555372 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9"} err="failed to get container status \"6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9\": rpc error: code = NotFound desc = could not find container \"6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9\": container with ID starting with 6ef1299feffa7784c864b364a87261a2e05a8b6d312f25f5217194bb92a1b9a9 not found: ID does not exist" Jul 22 01:01:02.555431 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.555396 2562 scope.go:117] "RemoveContainer" containerID="e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c" Jul 22 01:01:02.555656 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:01:02.555624 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c\": container with ID starting with e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c not found: ID does not exist" containerID="e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c" Jul 22 01:01:02.555700 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.555666 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c"} err="failed to get container status \"e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c\": rpc error: code = NotFound desc = could not find container \"e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c\": container with ID starting with e9b2505e8b1022ed4cacd2f4dd3ca3117c99165146855f9380ec8f38f4fbe41c not found: ID does not exist" Jul 22 01:01:02.562980 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.562960 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx"] Jul 22 01:01:02.569471 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:02.569451 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b80956c0-openshift-default-5c57575cb5-zl7bx"] Jul 22 01:01:03.905303 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:03.905269 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af54c397-300d-4945-9440-e71003e21f39" path="/var/lib/kubelet/pods/af54c397-300d-4945-9440-e71003e21f39/volumes" Jul 22 01:01:06.893012 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:06.892973 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:01:14.599367 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:14.599337 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerID="3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b" exitCode=0 Jul 22 01:01:14.599824 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:14.599411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" event={"ID":"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402","Type":"ContainerDied","Data":"3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b"} Jul 22 01:01:14.600026 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:14.600009 2562 scope.go:117] "RemoveContainer" containerID="3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b" Jul 22 01:01:14.601089 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:14.601059 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:01:14.601150 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:14.601131 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:01:14.601191 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:14.601162 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:01:15.528662 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:15.528616 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:15.605551 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:15.605517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" event={"ID":"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402","Type":"ContainerStarted","Data":"524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3"} Jul 22 01:01:15.834395 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:15.834365 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:15.835509 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:15.835487 2562 patch_prober.go:28] interesting pod/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:01:15.835604 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:15.835545 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:01:16.835236 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:16.835201 2562 patch_prober.go:28] interesting pod/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:01:16.835633 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:16.835257 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:01:17.835442 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:17.835360 2562 patch_prober.go:28] interesting pod/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:01:17.835442 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:17.835424 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:01:18.834764 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:18.834725 2562 patch_prober.go:28] interesting pod/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:01:18.834945 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:18.834810 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:01:19.835121 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:19.835087 2562 patch_prober.go:28] interesting pod/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:01:19.835507 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:19.835180 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:01:20.835443 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:20.835403 2562 patch_prober.go:28] interesting pod/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:01:20.835922 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:20.835465 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:01:21.834783 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:21.834745 2562 patch_prober.go:28] interesting pod/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 22 01:01:21.834963 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:21.834806 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 22 01:01:22.838398 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:22.838370 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:22.838805 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:22.838670 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:22.839381 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:22.839363 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:38.517298 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:38.517264 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:01:40.070348 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:40.070309 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz"] Jul 22 01:01:40.070822 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:40.070501 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" containerID="cri-o://524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3" gracePeriod=30 Jul 22 01:01:45.332355 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.332329 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:45.490445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490359 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gngc\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-kube-api-access-7gngc\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490406 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-envoy\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490442 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istiod-ca-cert\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490763 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490465 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-data\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490763 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490481 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-token\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490763 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490510 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-podinfo\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490763 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490554 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-certs\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490763 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490585 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-credential-socket\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.490763 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490619 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-socket\") pod \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\" (UID: \"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402\") " Jul 22 01:01:45.491072 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.490933 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:45.491072 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.491005 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:01:45.491180 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.491109 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:45.491235 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.491175 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:45.491569 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.491538 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:45.492765 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.492730 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:01:45.492870 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.492787 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-kube-api-access-7gngc" (OuterVolumeSpecName: "kube-api-access-7gngc") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "kube-api-access-7gngc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:01:45.492917 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.492866 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:01:45.492954 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.492915 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" (UID: "1c625cf3-7fe4-4846-bb7f-8ffed3f2e402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:01:45.591575 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591536 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591575 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591568 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591575 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591577 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591575 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591585 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591593 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591602 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591610 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gngc\" (UniqueName: \"kubernetes.io/projected/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-kube-api-access-7gngc\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591618 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.591871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.591626 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:01:45.747425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.747330 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerID="524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3" exitCode=0 Jul 22 01:01:45.747425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.747404 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" Jul 22 01:01:45.747611 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.747405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" event={"ID":"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402","Type":"ContainerDied","Data":"524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3"} Jul 22 01:01:45.747611 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.747562 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz" event={"ID":"1c625cf3-7fe4-4846-bb7f-8ffed3f2e402","Type":"ContainerDied","Data":"18dc17583e95723cd33de5804f87b2b2bd0f8322d56e8585d01280693519fec3"} Jul 22 01:01:45.747611 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.747588 2562 scope.go:117] "RemoveContainer" containerID="524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3" Jul 22 01:01:45.756332 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.756309 2562 scope.go:117] "RemoveContainer" containerID="3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b" Jul 22 01:01:45.763989 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.763973 2562 scope.go:117] "RemoveContainer" containerID="524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3" Jul 22 01:01:45.764223 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:01:45.764202 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3\": container with ID starting with 524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3 not found: ID does not exist" containerID="524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3" Jul 22 01:01:45.764263 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.764232 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3"} err="failed to get container status \"524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3\": rpc error: code = NotFound desc = could not find container \"524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3\": container with ID starting with 524b3714bf770e2ba7c74b8ff186d687446fe9987efeab2f3f5f124301f797c3 not found: ID does not exist" Jul 22 01:01:45.764263 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.764248 2562 scope.go:117] "RemoveContainer" containerID="3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b" Jul 22 01:01:45.764488 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:01:45.764471 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b\": container with ID starting with 3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b not found: ID does not exist" containerID="3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b" Jul 22 01:01:45.764545 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.764492 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b"} err="failed to get container status \"3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b\": rpc error: code = NotFound desc = could not find container \"3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b\": container with ID starting with 3405f9bab16f84325168b871bac41aef68cd0060b43ec35cab8838a51656193b not found: ID does not exist" Jul 22 01:01:45.770558 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.770538 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz"] Jul 22 01:01:45.773047 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.773029 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5186cc87-openshift-default-6db96b77c5-dtdnz"] Jul 22 01:01:45.905464 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:45.905432 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" path="/var/lib/kubelet/pods/1c625cf3-7fe4-4846-bb7f-8ffed3f2e402/volumes" Jul 22 01:01:55.951121 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:55.951091 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:01:55.955016 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:01:55.954988 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:03:08.228663 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:08.228561 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:03:18.789221 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:18.789184 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:03:23.125059 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125019 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq"] Jul 22 01:03:23.125587 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125567 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" Jul 22 01:03:23.125704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125590 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" Jul 22 01:03:23.125704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125606 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" Jul 22 01:03:23.125704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125614 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" Jul 22 01:03:23.125704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125636 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" Jul 22 01:03:23.125704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125661 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" Jul 22 01:03:23.125704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125679 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" Jul 22 01:03:23.125704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125687 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" Jul 22 01:03:23.126052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125787 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" Jul 22 01:03:23.126052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125801 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" Jul 22 01:03:23.126052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.125812 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c625cf3-7fe4-4846-bb7f-8ffed3f2e402" containerName="istio-proxy" Jul 22 01:03:23.128930 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.128910 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.133288 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.133269 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3a8f56da-openshift-default-dockercfg-ssz65\"" Jul 22 01:03:23.152615 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.152589 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq"] Jul 22 01:03:23.248706 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.248637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-token\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.248706 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.248715 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-data\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.248978 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.248734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-podinfo\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.248978 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.248846 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istiod-ca-cert\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.248978 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.248961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-credential-socket\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.249139 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.248987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-certs\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.249139 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.249034 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gd6q\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-kube-api-access-5gd6q\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.249139 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.249060 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-socket\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.249139 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.249098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-envoy\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350086 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gd6q\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-kube-api-access-5gd6q\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350250 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-socket\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350250 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-envoy\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350250 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-token\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350250 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-data\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350250 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-podinfo\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istiod-ca-cert\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-credential-socket\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-certs\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350686 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-socket\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350686 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-data\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350782 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350735 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-certs\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.350889 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.350868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-credential-socket\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.351196 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.351174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istiod-ca-cert\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.352555 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.352530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-envoy\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.352685 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.352667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-podinfo\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.363942 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.363915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-token\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.365152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.365135 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gd6q\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-kube-api-access-5gd6q\") pod \"e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.438917 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.438850 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:23.574627 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.574578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq"] Jul 22 01:03:23.577157 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:03:23.577131 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa16ad7_d41b_4696_83f0_d8da697c38d8.slice/crio-b039b36c4944e2826153634403267c2f53f5e78269f42e52aa5c06ede24ee597 WatchSource:0}: Error finding container b039b36c4944e2826153634403267c2f53f5e78269f42e52aa5c06ede24ee597: Status 404 returned error can't find the container with id b039b36c4944e2826153634403267c2f53f5e78269f42e52aa5c06ede24ee597 Jul 22 01:03:23.579248 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.579219 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:03:23.579329 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.579285 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:03:23.579329 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:23.579315 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:03:24.201990 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:24.201953 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" event={"ID":"4fa16ad7-d41b-4696-83f0-d8da697c38d8","Type":"ContainerStarted","Data":"57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d"} Jul 22 01:03:24.202378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:24.202014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" event={"ID":"4fa16ad7-d41b-4696-83f0-d8da697c38d8","Type":"ContainerStarted","Data":"b039b36c4944e2826153634403267c2f53f5e78269f42e52aa5c06ede24ee597"} Jul 22 01:03:24.224326 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:24.224276 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podStartSLOduration=1.224261364 podStartE2EDuration="1.224261364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:03:24.223819863 +0000 UTC m=+2188.854052415" watchObservedRunningTime="2026-07-22 01:03:24.224261364 +0000 UTC m=+2188.854493905" Jul 22 01:03:24.440023 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:24.439984 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:24.444993 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:24.444969 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:25.207102 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:25.207072 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:25.208194 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:25.208179 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:30.089123 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:30.089088 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:03:38.269933 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:38.269832 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerID="57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d" exitCode=0 Jul 22 01:03:38.269933 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:38.269902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" event={"ID":"4fa16ad7-d41b-4696-83f0-d8da697c38d8","Type":"ContainerDied","Data":"57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d"} Jul 22 01:03:38.270385 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:38.270303 2562 scope.go:117] "RemoveContainer" containerID="57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d" Jul 22 01:03:38.271286 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:38.271257 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:03:38.271360 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:38.271342 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:03:38.271411 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:38.271397 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:03:39.277013 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:39.276979 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" event={"ID":"4fa16ad7-d41b-4696-83f0-d8da697c38d8","Type":"ContainerStarted","Data":"203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8"} Jul 22 01:03:39.439582 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:39.439546 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:39.440732 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:39.440706 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 01:03:39.440851 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:39.440755 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 01:03:40.207767 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:40.207727 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:40.439834 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:40.439800 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 01:03:40.440212 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:40.439860 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 01:03:41.439599 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:41.439564 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 01:03:41.439792 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:41.439633 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 01:03:42.439441 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:42.439401 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 01:03:42.439864 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:42.439463 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 01:03:43.440116 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:43.440080 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 01:03:43.440515 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:43.440135 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 01:03:44.439449 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:44.439410 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 01:03:44.439622 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:44.439466 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 01:03:45.439728 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:45.439694 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 22 01:03:45.440119 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:45.439759 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 22 01:03:46.442849 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:46.442817 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:46.443739 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:46.443720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:03:51.749151 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:51.749113 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:03:59.796721 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:59.796686 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq"] Jul 22 01:03:59.797203 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:03:59.796901 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" containerID="cri-o://203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8" gracePeriod=30 Jul 22 01:04:01.445274 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:01.445231 2562 patch_prober.go:28] interesting pod/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 22 01:04:01.445695 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:01.445303 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 01:04:05.052222 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.052198 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:04:05.129307 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129234 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-data\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129307 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129302 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gd6q\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-kube-api-access-5gd6q\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129499 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129328 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-socket\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129499 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129366 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istiod-ca-cert\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129499 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129409 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-token\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129499 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129444 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-credential-socket\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129726 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129499 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-certs\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129726 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-envoy\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129726 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-podinfo\") pod \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\" (UID: \"4fa16ad7-d41b-4696-83f0-d8da697c38d8\") " Jul 22 01:04:05.129726 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129614 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:04:05.129948 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129800 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:04:05.129948 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.129821 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:04:05.130052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.130027 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.130052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.130046 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.130154 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.130063 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.130154 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.130061 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:04:05.130281 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.130254 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:04:05.131690 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.131665 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:04:05.131788 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.131712 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:04:05.131788 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.131753 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:04:05.131872 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.131786 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-kube-api-access-5gd6q" (OuterVolumeSpecName: "kube-api-access-5gd6q") pod "4fa16ad7-d41b-4696-83f0-d8da697c38d8" (UID: "4fa16ad7-d41b-4696-83f0-d8da697c38d8"). InnerVolumeSpecName "kube-api-access-5gd6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:04:05.230719 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.230686 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.230719 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.230713 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gd6q\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-kube-api-access-5gd6q\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.230719 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.230725 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.230941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.230735 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.230941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.230745 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.230941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.230752 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa16ad7-d41b-4696-83f0-d8da697c38d8-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:04:05.401825 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.401735 2562 generic.go:356] "Generic (PLEG): container finished" podID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerID="203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8" exitCode=0 Jul 22 01:04:05.401825 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.401793 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" Jul 22 01:04:05.401990 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.401823 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" event={"ID":"4fa16ad7-d41b-4696-83f0-d8da697c38d8","Type":"ContainerDied","Data":"203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8"} Jul 22 01:04:05.401990 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.401883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq" event={"ID":"4fa16ad7-d41b-4696-83f0-d8da697c38d8","Type":"ContainerDied","Data":"b039b36c4944e2826153634403267c2f53f5e78269f42e52aa5c06ede24ee597"} Jul 22 01:04:05.401990 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.401914 2562 scope.go:117] "RemoveContainer" containerID="203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8" Jul 22 01:04:05.410566 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.410548 2562 scope.go:117] "RemoveContainer" containerID="57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d" Jul 22 01:04:05.418051 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.418031 2562 scope.go:117] "RemoveContainer" containerID="203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8" Jul 22 01:04:05.418317 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:04:05.418298 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8\": container with ID starting with 203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8 not found: ID does not exist" containerID="203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8" Jul 22 01:04:05.418370 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.418325 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8"} err="failed to get container status \"203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8\": rpc error: code = NotFound desc = could not find container \"203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8\": container with ID starting with 203da2badbd40f9e4410c825254e3f69c5eb436d99a74baedbdad689f01336a8 not found: ID does not exist" Jul 22 01:04:05.418370 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.418343 2562 scope.go:117] "RemoveContainer" containerID="57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d" Jul 22 01:04:05.418572 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:04:05.418554 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d\": container with ID starting with 57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d not found: ID does not exist" containerID="57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d" Jul 22 01:04:05.418609 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.418577 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d"} err="failed to get container status \"57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d\": rpc error: code = NotFound desc = could not find container \"57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d\": container with ID starting with 57102f2321622794dba516a4eefb139242d437077046b829928184b99d0a796d not found: ID does not exist" Jul 22 01:04:05.422802 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.422780 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq"] Jul 22 01:04:05.427467 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.427442 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3a8f56da-openshift-default-7c994cfdf5-lv5kq"] Jul 22 01:04:05.908593 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:05.908557 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" path="/var/lib/kubelet/pods/4fa16ad7-d41b-4696-83f0-d8da697c38d8/volumes" Jul 22 01:04:19.634626 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.634540 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s"] Jul 22 01:04:19.635049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.634982 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" Jul 22 01:04:19.635049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.634995 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" Jul 22 01:04:19.635049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.635016 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" Jul 22 01:04:19.635049 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.635022 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" Jul 22 01:04:19.635182 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.635096 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af54c397-300d-4945-9440-e71003e21f39" containerName="istio-proxy" Jul 22 01:04:19.635182 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.635109 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" Jul 22 01:04:19.635182 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.635118 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa16ad7-d41b-4696-83f0-d8da697c38d8" containerName="istio-proxy" Jul 22 01:04:19.639608 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.639589 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.641873 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.641850 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c9532535-openshift-default-dockercfg-rbmnp\"" Jul 22 01:04:19.651476 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.651452 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s"] Jul 22 01:04:19.754403 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-data\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754403 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-istio-token\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754660 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754445 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-socket\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754660 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08655467-da3e-46b1-a874-747e63ee7007-istiod-ca-cert\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754782 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754636 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-credential-socket\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754782 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754699 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-envoy\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754782 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-certs\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754782 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08655467-da3e-46b1-a874-747e63ee7007-istio-podinfo\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.754922 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.754801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fv9n\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-kube-api-access-5fv9n\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.855744 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.855713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-data\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.855744 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.855746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-istio-token\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.855983 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.855784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-socket\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.855983 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.855907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08655467-da3e-46b1-a874-747e63ee7007-istiod-ca-cert\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856108 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.855978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-credential-socket\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856108 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-envoy\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856108 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-certs\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856108 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856078 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08655467-da3e-46b1-a874-747e63ee7007-istio-podinfo\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856108 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fv9n\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-kube-api-access-5fv9n\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856350 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856168 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-socket\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856350 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-data\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856350 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856329 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-credential-socket\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856630 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-certs\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.856750 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.856682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08655467-da3e-46b1-a874-747e63ee7007-istiod-ca-cert\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.858447 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.858412 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-envoy\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.858630 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.858614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08655467-da3e-46b1-a874-747e63ee7007-istio-podinfo\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.865090 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.865067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fv9n\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-kube-api-access-5fv9n\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.865207 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.865070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-istio-token\") pod \"e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:19.951003 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:19.950916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:20.089613 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.089588 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s"] Jul 22 01:04:20.091917 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:04:20.091886 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08655467_da3e_46b1_a874_747e63ee7007.slice/crio-862dd90a8a3936c24d9ac23d56030658ca87e7c11e3eac3a5a9b37257a19cb77 WatchSource:0}: Error finding container 862dd90a8a3936c24d9ac23d56030658ca87e7c11e3eac3a5a9b37257a19cb77: Status 404 returned error can't find the container with id 862dd90a8a3936c24d9ac23d56030658ca87e7c11e3eac3a5a9b37257a19cb77 Jul 22 01:04:20.094479 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.094447 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:04:20.094541 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.094522 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:04:20.094577 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.094555 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:04:20.470279 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.470242 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" event={"ID":"08655467-da3e-46b1-a874-747e63ee7007","Type":"ContainerStarted","Data":"fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c"} Jul 22 01:04:20.470459 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.470331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" event={"ID":"08655467-da3e-46b1-a874-747e63ee7007","Type":"ContainerStarted","Data":"862dd90a8a3936c24d9ac23d56030658ca87e7c11e3eac3a5a9b37257a19cb77"} Jul 22 01:04:20.492271 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.492217 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" podStartSLOduration=1.4922025159999999 podStartE2EDuration="1.492202516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:04:20.490455161 +0000 UTC m=+2245.120687701" watchObservedRunningTime="2026-07-22 01:04:20.492202516 +0000 UTC m=+2245.122435057" Jul 22 01:04:20.951748 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:20.951713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:21.956784 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:21.956754 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:22.481900 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:22.481858 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:22.482913 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:22.482889 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:29.485742 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:29.485709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:04:37.483504 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.483462 2562 patch_prober.go:28] interesting pod/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 01:04:37.483943 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.483533 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 01:04:37.550847 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.550817 2562 generic.go:356] "Generic (PLEG): container finished" podID="08655467-da3e-46b1-a874-747e63ee7007" containerID="fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c" exitCode=0 Jul 22 01:04:37.551034 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.550895 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" event={"ID":"08655467-da3e-46b1-a874-747e63ee7007","Type":"ContainerDied","Data":"fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c"} Jul 22 01:04:37.551334 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.551317 2562 scope.go:117] "RemoveContainer" containerID="fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c" Jul 22 01:04:37.552378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.552347 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:04:37.552479 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.552433 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:04:37.552542 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.552477 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:04:37.795267 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:37.789660 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:04:38.561670 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:38.561466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" event={"ID":"08655467-da3e-46b1-a874-747e63ee7007","Type":"ContainerStarted","Data":"fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d"} Jul 22 01:04:38.951191 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:38.951091 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:38.951367 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:38.951336 2562 patch_prober.go:28] interesting pod/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 01:04:38.951438 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:38.951399 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 01:04:39.951764 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:39.951530 2562 patch_prober.go:28] interesting pod/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 01:04:39.951764 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:39.951611 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 01:04:40.952262 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:40.952212 2562 patch_prober.go:28] interesting pod/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 01:04:40.952785 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:40.952295 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 01:04:41.952288 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:41.952246 2562 patch_prober.go:28] interesting pod/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 22 01:04:41.952807 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:41.952316 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 22 01:04:42.954880 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:42.954849 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:42.955253 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:42.955186 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:42.955794 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:42.955772 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:04:51.477680 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:04:51.477627 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:05:09.384462 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:05:09.384424 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:05:18.071236 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:05:18.071200 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:06:05.298797 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:05.298707 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s"] Jul 22 01:06:05.299341 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:05.298972 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" containerID="cri-o://fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d" gracePeriod=30 Jul 22 01:06:09.283257 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:09.283221 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:06:10.553459 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.553428 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:06:10.638154 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638070 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08655467-da3e-46b1-a874-747e63ee7007-istio-podinfo\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638154 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638111 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fv9n\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-kube-api-access-5fv9n\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638154 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638155 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-data\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638428 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638176 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-istio-token\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638428 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638199 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-socket\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638428 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638219 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08655467-da3e-46b1-a874-747e63ee7007-istiod-ca-cert\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638428 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638237 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-certs\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638428 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638305 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-credential-socket\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638428 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638348 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-envoy\") pod \"08655467-da3e-46b1-a874-747e63ee7007\" (UID: \"08655467-da3e-46b1-a874-747e63ee7007\") " Jul 22 01:06:10.638765 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638515 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:06:10.638765 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638530 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:06:10.638765 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638601 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08655467-da3e-46b1-a874-747e63ee7007-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:06:10.638765 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638673 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.638765 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638738 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.639030 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.638775 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:06:10.639362 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.639340 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-data" (OuterVolumeSpecName: "istio-data") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:06:10.640445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.640414 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08655467-da3e-46b1-a874-747e63ee7007-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:06:10.640546 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.640414 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:06:10.640754 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.640738 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-istio-token" (OuterVolumeSpecName: "istio-token") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:06:10.640838 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.640787 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-kube-api-access-5fv9n" (OuterVolumeSpecName: "kube-api-access-5fv9n") pod "08655467-da3e-46b1-a874-747e63ee7007" (UID: "08655467-da3e-46b1-a874-747e63ee7007"). InnerVolumeSpecName "kube-api-access-5fv9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:06:10.739271 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.739244 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08655467-da3e-46b1-a874-747e63ee7007-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.739271 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.739270 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fv9n\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-kube-api-access-5fv9n\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.739271 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.739281 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.739513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.739290 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08655467-da3e-46b1-a874-747e63ee7007-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.739513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.739298 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08655467-da3e-46b1-a874-747e63ee7007-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.739513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.739306 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.739513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.739313 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08655467-da3e-46b1-a874-747e63ee7007-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:06:10.993437 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.993354 2562 generic.go:356] "Generic (PLEG): container finished" podID="08655467-da3e-46b1-a874-747e63ee7007" containerID="fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d" exitCode=0 Jul 22 01:06:10.993437 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.993403 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" Jul 22 01:06:10.993629 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.993437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" event={"ID":"08655467-da3e-46b1-a874-747e63ee7007","Type":"ContainerDied","Data":"fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d"} Jul 22 01:06:10.993629 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.993504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s" event={"ID":"08655467-da3e-46b1-a874-747e63ee7007","Type":"ContainerDied","Data":"862dd90a8a3936c24d9ac23d56030658ca87e7c11e3eac3a5a9b37257a19cb77"} Jul 22 01:06:10.993629 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:10.993529 2562 scope.go:117] "RemoveContainer" containerID="fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d" Jul 22 01:06:11.002928 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.002910 2562 scope.go:117] "RemoveContainer" containerID="fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c" Jul 22 01:06:11.010982 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.010965 2562 scope.go:117] "RemoveContainer" containerID="fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d" Jul 22 01:06:11.011243 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:06:11.011224 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d\": container with ID starting with fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d not found: ID does not exist" containerID="fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d" Jul 22 01:06:11.011310 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.011252 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d"} err="failed to get container status \"fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d\": rpc error: code = NotFound desc = could not find container \"fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d\": container with ID starting with fe4b11c45e0f9dfbfd92b75de1565303fecbf0266c7aca3976c0965b94c1e35d not found: ID does not exist" Jul 22 01:06:11.011310 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.011269 2562 scope.go:117] "RemoveContainer" containerID="fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c" Jul 22 01:06:11.011490 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:06:11.011472 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c\": container with ID starting with fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c not found: ID does not exist" containerID="fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c" Jul 22 01:06:11.011531 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.011496 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c"} err="failed to get container status \"fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c\": rpc error: code = NotFound desc = could not find container \"fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c\": container with ID starting with fe968eb57d2d1e1b79fb3f4110fda5b421dcac6714e76fd7045f5b76a7eb0a9c not found: ID does not exist" Jul 22 01:06:11.014297 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.014277 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s"] Jul 22 01:06:11.018070 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.018049 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c9532535-openshift-default-6f6685b9f-ddn4s"] Jul 22 01:06:11.907522 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:11.907484 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08655467-da3e-46b1-a874-747e63ee7007" path="/var/lib/kubelet/pods/08655467-da3e-46b1-a874-747e63ee7007/volumes" Jul 22 01:06:20.882140 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:20.882105 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:06:55.978232 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:55.978203 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:06:55.983372 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:06:55.983350 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:07:00.299925 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.299891 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc"] Jul 22 01:07:00.300298 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.300242 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" Jul 22 01:07:00.300298 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.300253 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" Jul 22 01:07:00.300298 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.300263 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" Jul 22 01:07:00.300298 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.300271 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" Jul 22 01:07:00.300425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.300341 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" Jul 22 01:07:00.300425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.300351 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="08655467-da3e-46b1-a874-747e63ee7007" containerName="istio-proxy" Jul 22 01:07:00.304797 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.304778 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.307244 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.307222 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-805a5b-openshift-default-dockercfg-jlkf7\"" Jul 22 01:07:00.318194 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.318166 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc"] Jul 22 01:07:00.372869 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.372841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-envoy\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373048 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.372875 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-credential-socket\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373048 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.372962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-socket\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373048 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.373003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-certs\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373223 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.373054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-data\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373223 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.373090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1efc198-30e4-414b-94d4-20c8351cde71-istiod-ca-cert\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373223 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.373133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-istio-token\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373223 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.373162 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5djh\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-kube-api-access-x5djh\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.373223 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.373198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1efc198-30e4-414b-94d4-20c8351cde71-istio-podinfo\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474126 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-socket\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474126 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-certs\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-data\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1efc198-30e4-414b-94d4-20c8351cde71-istiod-ca-cert\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-istio-token\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5djh\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-kube-api-access-x5djh\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1efc198-30e4-414b-94d4-20c8351cde71-istio-podinfo\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474676 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-envoy\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474676 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474486 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-certs\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474676 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-credential-socket\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474676 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-socket\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474676 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-data\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.474851 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474758 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-credential-socket\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.475005 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.474980 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1efc198-30e4-414b-94d4-20c8351cde71-istiod-ca-cert\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.476631 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.476612 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-envoy\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.476912 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.476894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1efc198-30e4-414b-94d4-20c8351cde71-istio-podinfo\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.483314 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.483288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-istio-token\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.483419 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.483320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5djh\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-kube-api-access-x5djh\") pod \"e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.615394 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.615306 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:00.751156 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.751118 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc"] Jul 22 01:07:00.754159 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:07:00.754128 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1efc198_30e4_414b_94d4_20c8351cde71.slice/crio-de7a323f5052a8c3f046e810cd5fbc48c85fd265334551b2c1469deb55f7e14b WatchSource:0}: Error finding container de7a323f5052a8c3f046e810cd5fbc48c85fd265334551b2c1469deb55f7e14b: Status 404 returned error can't find the container with id de7a323f5052a8c3f046e810cd5fbc48c85fd265334551b2c1469deb55f7e14b Jul 22 01:07:00.756401 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.756364 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:00.756475 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.756449 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:00.756513 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:00.756493 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:01.228931 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:01.228897 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" event={"ID":"d1efc198-30e4-414b-94d4-20c8351cde71","Type":"ContainerStarted","Data":"809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1"} Jul 22 01:07:01.229098 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:01.228949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" event={"ID":"d1efc198-30e4-414b-94d4-20c8351cde71","Type":"ContainerStarted","Data":"de7a323f5052a8c3f046e810cd5fbc48c85fd265334551b2c1469deb55f7e14b"} Jul 22 01:07:01.249162 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:01.249097 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" podStartSLOduration=1.249074483 podStartE2EDuration="1.249074483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:07:01.247209523 +0000 UTC m=+2405.877442064" watchObservedRunningTime="2026-07-22 01:07:01.249074483 +0000 UTC m=+2405.879307024" Jul 22 01:07:01.616252 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:01.616213 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:01.621057 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:01.621029 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:02.233981 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:02.233940 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:02.234916 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:02.234892 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:08.298587 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.298556 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8"] Jul 22 01:07:08.301855 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.301837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.304209 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.304186 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-75fe9f-openshift-default-dockercfg-b24j6\"" Jul 22 01:07:08.316410 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.316386 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8"] Jul 22 01:07:08.343483 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-envoy\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343619 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-socket\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343619 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-certs\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343619 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8gq\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-kube-api-access-6k8gq\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343619 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-podinfo\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-token\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-credential-socket\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343721 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-data\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.343919 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.343812 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7ebb11-4e78-4c5a-812e-53e23404727c-istiod-ca-cert\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.444954 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.444916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7ebb11-4e78-4c5a-812e-53e23404727c-istiod-ca-cert\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445138 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.444965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-envoy\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445138 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-socket\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445138 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-certs\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445138 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8gq\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-kube-api-access-6k8gq\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445138 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-podinfo\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445138 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-token\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445387 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-credential-socket\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445387 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-data\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445716 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445687 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-credential-socket\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445825 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-certs\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445825 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-socket\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445825 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7ebb11-4e78-4c5a-812e-53e23404727c-istiod-ca-cert\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.445947 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.445802 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-data\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.447757 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.447738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-envoy\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.447878 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.447850 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-podinfo\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.453557 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.453534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-token\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.453704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.453591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8gq\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-kube-api-access-6k8gq\") pod \"e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.611696 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.611594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:08.759826 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.759803 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8"] Jul 22 01:07:08.762335 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:07:08.762308 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7ebb11_4e78_4c5a_812e_53e23404727c.slice/crio-c841dd0fe0d0ea303a7ea639bd42df070e3092b28e0946d64f36376470b1b13d WatchSource:0}: Error finding container c841dd0fe0d0ea303a7ea639bd42df070e3092b28e0946d64f36376470b1b13d: Status 404 returned error can't find the container with id c841dd0fe0d0ea303a7ea639bd42df070e3092b28e0946d64f36376470b1b13d Jul 22 01:07:08.764352 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.764321 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:08.764408 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.764386 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:08.764444 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.764415 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:08.786344 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:08.786321 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:07:09.267055 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:09.267025 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" event={"ID":"ec7ebb11-4e78-4c5a-812e-53e23404727c","Type":"ContainerStarted","Data":"33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37"} Jul 22 01:07:09.267226 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:09.267074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" event={"ID":"ec7ebb11-4e78-4c5a-812e-53e23404727c","Type":"ContainerStarted","Data":"c841dd0fe0d0ea303a7ea639bd42df070e3092b28e0946d64f36376470b1b13d"} Jul 22 01:07:09.293564 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:09.293510 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" podStartSLOduration=1.293488806 podStartE2EDuration="1.293488806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:07:09.291968289 +0000 UTC m=+2413.922200830" watchObservedRunningTime="2026-07-22 01:07:09.293488806 +0000 UTC m=+2413.923721348" Jul 22 01:07:09.611729 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:09.611691 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:09.617218 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:09.617192 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:10.274189 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:10.274146 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:10.275233 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:10.275202 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:17.235739 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.235696 2562 patch_prober.go:28] interesting pod/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 01:07:17.236160 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.235752 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 01:07:17.310365 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.310333 2562 generic.go:356] "Generic (PLEG): container finished" podID="d1efc198-30e4-414b-94d4-20c8351cde71" containerID="809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1" exitCode=0 Jul 22 01:07:17.310528 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.310377 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" event={"ID":"d1efc198-30e4-414b-94d4-20c8351cde71","Type":"ContainerDied","Data":"809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1"} Jul 22 01:07:17.310780 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.310765 2562 scope.go:117] "RemoveContainer" containerID="809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1" Jul 22 01:07:17.311840 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.311808 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:17.311945 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.311877 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:17.311945 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:17.311913 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:18.317043 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:18.317010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" event={"ID":"d1efc198-30e4-414b-94d4-20c8351cde71","Type":"ContainerStarted","Data":"78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55"} Jul 22 01:07:18.615763 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:18.615680 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:18.615942 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:18.615917 2562 patch_prober.go:28] interesting pod/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 01:07:18.616023 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:18.616004 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 01:07:19.616279 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:19.616241 2562 patch_prober.go:28] interesting pod/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 01:07:19.616627 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:19.616313 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 01:07:20.616089 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:20.616049 2562 patch_prober.go:28] interesting pod/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 01:07:20.616290 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:20.616111 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 01:07:20.979242 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:20.979145 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:07:21.616743 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:21.616697 2562 patch_prober.go:28] interesting pod/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 22 01:07:21.617137 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:21.616760 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 22 01:07:22.620125 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:22.620089 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:22.620553 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:22.620422 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:22.621052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:22.621034 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:07:23.342202 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:23.342170 2562 generic.go:356] "Generic (PLEG): container finished" podID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerID="33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37" exitCode=0 Jul 22 01:07:23.342427 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:23.342245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" event={"ID":"ec7ebb11-4e78-4c5a-812e-53e23404727c","Type":"ContainerDied","Data":"33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37"} Jul 22 01:07:23.342766 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:23.342746 2562 scope.go:117] "RemoveContainer" containerID="33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37" Jul 22 01:07:23.343725 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:23.343696 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:23.343803 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:23.343785 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:23.343854 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:23.343837 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:07:24.352687 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:24.352619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" event={"ID":"ec7ebb11-4e78-4c5a-812e-53e23404727c","Type":"ContainerStarted","Data":"1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698"} Jul 22 01:07:24.611813 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:24.611714 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:24.613012 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:24.612986 2562 patch_prober.go:28] interesting pod/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 01:07:24.613140 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:24.613050 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 01:07:25.274299 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:25.274256 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:25.612387 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:25.612352 2562 patch_prober.go:28] interesting pod/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 01:07:25.612821 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:25.612403 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 01:07:26.612403 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:26.612368 2562 patch_prober.go:28] interesting pod/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 01:07:26.612968 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:26.612428 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 01:07:27.612005 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:27.611964 2562 patch_prober.go:28] interesting pod/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 22 01:07:27.612270 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:27.612026 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 22 01:07:28.615658 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:28.615616 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:28.616698 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:28.616676 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:07:29.882291 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:07:29.882257 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:08:01.578813 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:08:01.578776 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:08:40.680390 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:08:40.680355 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:08:54.482628 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:08:54.482532 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:08:58.438771 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:08:58.438736 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc"] Jul 22 01:08:58.439197 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:08:58.438989 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" containerID="cri-o://78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55" gracePeriod=30 Jul 22 01:09:03.689544 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.689517 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:09:03.728825 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.728750 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1efc198-30e4-414b-94d4-20c8351cde71-istiod-ca-cert\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.728825 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.728794 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-socket\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.728831 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-data\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.728848 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-certs\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.728863 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1efc198-30e4-414b-94d4-20c8351cde71-istio-podinfo\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.728915 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-istio-token\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.728971 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-credential-socket\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729001 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5djh\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-kube-api-access-x5djh\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729030 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-envoy\") pod \"d1efc198-30e4-414b-94d4-20c8351cde71\" (UID: \"d1efc198-30e4-414b-94d4-20c8351cde71\") " Jul 22 01:09:03.729378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729224 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:03.729378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729278 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1efc198-30e4-414b-94d4-20c8351cde71-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:09:03.729378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729283 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:03.729378 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729365 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1efc198-30e4-414b-94d4-20c8351cde71-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.729615 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729411 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.729615 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729455 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:03.729758 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.729703 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:03.731425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.731388 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1efc198-30e4-414b-94d4-20c8351cde71-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:09:03.731527 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.731419 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:03.731707 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.731683 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:09:03.731773 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.731703 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-kube-api-access-x5djh" (OuterVolumeSpecName: "kube-api-access-x5djh") pod "d1efc198-30e4-414b-94d4-20c8351cde71" (UID: "d1efc198-30e4-414b-94d4-20c8351cde71"). InnerVolumeSpecName "kube-api-access-x5djh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:09:03.814909 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.814880 2562 generic.go:356] "Generic (PLEG): container finished" podID="d1efc198-30e4-414b-94d4-20c8351cde71" containerID="78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55" exitCode=0 Jul 22 01:09:03.815055 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.814928 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" Jul 22 01:09:03.815055 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.814963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" event={"ID":"d1efc198-30e4-414b-94d4-20c8351cde71","Type":"ContainerDied","Data":"78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55"} Jul 22 01:09:03.815055 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.815016 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc" event={"ID":"d1efc198-30e4-414b-94d4-20c8351cde71","Type":"ContainerDied","Data":"de7a323f5052a8c3f046e810cd5fbc48c85fd265334551b2c1469deb55f7e14b"} Jul 22 01:09:03.815055 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.815042 2562 scope.go:117] "RemoveContainer" containerID="78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55" Jul 22 01:09:03.823942 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.823927 2562 scope.go:117] "RemoveContainer" containerID="809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1" Jul 22 01:09:03.830696 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.830676 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.830696 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.830695 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.830858 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.830705 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1efc198-30e4-414b-94d4-20c8351cde71-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.830858 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.830713 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.830858 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.830721 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.830858 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.830731 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5djh\" (UniqueName: \"kubernetes.io/projected/d1efc198-30e4-414b-94d4-20c8351cde71-kube-api-access-x5djh\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.830858 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.830740 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1efc198-30e4-414b-94d4-20c8351cde71-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:03.831756 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.831743 2562 scope.go:117] "RemoveContainer" containerID="78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55" Jul 22 01:09:03.832001 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:09:03.831983 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55\": container with ID starting with 78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55 not found: ID does not exist" containerID="78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55" Jul 22 01:09:03.832041 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.832008 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55"} err="failed to get container status \"78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55\": rpc error: code = NotFound desc = could not find container \"78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55\": container with ID starting with 78ef37f53eb32155989237bc624a47fd18427ae4b646107ec47a7bd172a9ab55 not found: ID does not exist" Jul 22 01:09:03.832041 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.832023 2562 scope.go:117] "RemoveContainer" containerID="809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1" Jul 22 01:09:03.832257 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:09:03.832239 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1\": container with ID starting with 809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1 not found: ID does not exist" containerID="809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1" Jul 22 01:09:03.832293 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.832264 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1"} err="failed to get container status \"809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1\": rpc error: code = NotFound desc = could not find container \"809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1\": container with ID starting with 809eaafdac3c7e31c04fd68a59852da91679c8ca98e1398c2efc75f952f4cdd1 not found: ID does not exist" Jul 22 01:09:03.852319 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.852300 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc"] Jul 22 01:09:03.864863 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.864844 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-805a5b-openshift-default-849f48d6dc-zmnhc"] Jul 22 01:09:03.905712 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:03.905686 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" path="/var/lib/kubelet/pods/d1efc198-30e4-414b-94d4-20c8351cde71/volumes" Jul 22 01:09:12.491824 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:12.491788 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:09:21.068977 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:21.068944 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8"] Jul 22 01:09:21.069493 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:21.069167 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" containerID="cri-o://1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698" gracePeriod=30 Jul 22 01:09:22.011621 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.011590 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh"] Jul 22 01:09:22.011958 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.011946 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" Jul 22 01:09:22.012002 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.011960 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" Jul 22 01:09:22.012002 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.011976 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" Jul 22 01:09:22.012002 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.011982 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" Jul 22 01:09:22.012094 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.012050 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" Jul 22 01:09:22.012094 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.012060 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1efc198-30e4-414b-94d4-20c8351cde71" containerName="istio-proxy" Jul 22 01:09:22.016500 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.016483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.019539 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.019520 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-659cfa-openshift-default-dockercfg-9nspw\"" Jul 22 01:09:22.029984 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.029963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh"] Jul 22 01:09:22.086486 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-credential-socket\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086486 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-envoy\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaba292-633a-40d9-b1e5-219701710c06-istio-podinfo\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-socket\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-certs\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-data\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-istio-token\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4hw\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-kube-api-access-vq4hw\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.086946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.086894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaba292-633a-40d9-b1e5-219701710c06-istiod-ca-cert\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.187590 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4hw\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-kube-api-access-vq4hw\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.187774 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaba292-633a-40d9-b1e5-219701710c06-istiod-ca-cert\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.187774 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-credential-socket\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.187774 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-envoy\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.187774 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaba292-633a-40d9-b1e5-219701710c06-istio-podinfo\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.187774 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-socket\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188046 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-certs\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188046 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-data\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188046 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.187917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-istio-token\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188210 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.188073 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-credential-socket\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188366 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.188320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-socket\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188505 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.188368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-data\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188505 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.188386 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-certs\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.188628 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.188548 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaba292-633a-40d9-b1e5-219701710c06-istiod-ca-cert\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.189992 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.189970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-envoy\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.190210 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.190194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaba292-633a-40d9-b1e5-219701710c06-istio-podinfo\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.197631 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.197607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-istio-token\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.197759 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.197736 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4hw\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-kube-api-access-vq4hw\") pod \"e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.326907 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.326875 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:22.464582 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.464529 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh"] Jul 22 01:09:22.467421 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:09:22.467384 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefaba292_633a_40d9_b1e5_219701710c06.slice/crio-c2732eea1fd08bfd24162ba16490a3b723579771b68d65c78254ee50d9aa72d0 WatchSource:0}: Error finding container c2732eea1fd08bfd24162ba16490a3b723579771b68d65c78254ee50d9aa72d0: Status 404 returned error can't find the container with id c2732eea1fd08bfd24162ba16490a3b723579771b68d65c78254ee50d9aa72d0 Jul 22 01:09:22.469635 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.469592 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:22.469737 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.469704 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:22.469784 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.469751 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:22.906478 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.906445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" event={"ID":"efaba292-633a-40d9-b1e5-219701710c06","Type":"ContainerStarted","Data":"1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989"} Jul 22 01:09:22.906712 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.906507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" event={"ID":"efaba292-633a-40d9-b1e5-219701710c06","Type":"ContainerStarted","Data":"c2732eea1fd08bfd24162ba16490a3b723579771b68d65c78254ee50d9aa72d0"} Jul 22 01:09:22.928090 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:22.928047 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podStartSLOduration=1.928032771 podStartE2EDuration="1.928032771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:09:22.926955618 +0000 UTC m=+2547.557188156" watchObservedRunningTime="2026-07-22 01:09:22.928032771 +0000 UTC m=+2547.558265311" Jul 22 01:09:23.327681 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:23.327627 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:23.332508 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:23.332484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:23.910892 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:23.910856 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:23.911811 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:23.911792 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:26.328700 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.328675 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:09:26.425922 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.425833 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-token\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.425922 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.425876 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-podinfo\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.425933 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-credential-socket\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.425958 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-certs\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.425985 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7ebb11-4e78-4c5a-812e-53e23404727c-istiod-ca-cert\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426072 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-socket\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426135 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8gq\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-kube-api-access-6k8gq\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426411 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426216 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-envoy\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426411 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426247 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-data\") pod \"ec7ebb11-4e78-4c5a-812e-53e23404727c\" (UID: \"ec7ebb11-4e78-4c5a-812e-53e23404727c\") " Jul 22 01:09:26.426411 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426289 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:26.426411 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426300 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:26.426411 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426366 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7ebb11-4e78-4c5a-812e-53e23404727c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:09:26.426677 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426580 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.426677 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426600 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.426677 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426614 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7ebb11-4e78-4c5a-812e-53e23404727c-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.426677 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.426671 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:26.427301 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.427277 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:26.428200 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.428169 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:09:26.428316 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.428244 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:09:26.428316 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.428255 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:09:26.428316 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.428248 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-kube-api-access-6k8gq" (OuterVolumeSpecName: "kube-api-access-6k8gq") pod "ec7ebb11-4e78-4c5a-812e-53e23404727c" (UID: "ec7ebb11-4e78-4c5a-812e-53e23404727c"). InnerVolumeSpecName "kube-api-access-6k8gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:09:26.527229 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.527198 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.527229 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.527227 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.527229 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.527237 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.527445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.527245 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8gq\" (UniqueName: \"kubernetes.io/projected/ec7ebb11-4e78-4c5a-812e-53e23404727c-kube-api-access-6k8gq\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.527445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.527255 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.527445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.527263 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7ebb11-4e78-4c5a-812e-53e23404727c-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:09:26.926060 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.926025 2562 generic.go:356] "Generic (PLEG): container finished" podID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerID="1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698" exitCode=0 Jul 22 01:09:26.926227 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.926075 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" Jul 22 01:09:26.926227 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.926103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" event={"ID":"ec7ebb11-4e78-4c5a-812e-53e23404727c","Type":"ContainerDied","Data":"1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698"} Jul 22 01:09:26.926227 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.926153 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8" event={"ID":"ec7ebb11-4e78-4c5a-812e-53e23404727c","Type":"ContainerDied","Data":"c841dd0fe0d0ea303a7ea639bd42df070e3092b28e0946d64f36376470b1b13d"} Jul 22 01:09:26.926227 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.926177 2562 scope.go:117] "RemoveContainer" containerID="1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698" Jul 22 01:09:26.935209 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.935193 2562 scope.go:117] "RemoveContainer" containerID="33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37" Jul 22 01:09:26.942601 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.942585 2562 scope.go:117] "RemoveContainer" containerID="1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698" Jul 22 01:09:26.942879 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:09:26.942860 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698\": container with ID starting with 1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698 not found: ID does not exist" containerID="1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698" Jul 22 01:09:26.942946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.942888 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698"} err="failed to get container status \"1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698\": rpc error: code = NotFound desc = could not find container \"1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698\": container with ID starting with 1395a66f244492be38649d9b3bc8881512567471640ec9f554ff7ce79ef8a698 not found: ID does not exist" Jul 22 01:09:26.942946 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.942904 2562 scope.go:117] "RemoveContainer" containerID="33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37" Jul 22 01:09:26.943100 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:09:26.943081 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37\": container with ID starting with 33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37 not found: ID does not exist" containerID="33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37" Jul 22 01:09:26.943145 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.943106 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37"} err="failed to get container status \"33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37\": rpc error: code = NotFound desc = could not find container \"33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37\": container with ID starting with 33c0ed03a86195151914ac87cc7eb8494104e86e67caf522fbcbac6c71986f37 not found: ID does not exist" Jul 22 01:09:26.953989 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.953967 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8"] Jul 22 01:09:26.962626 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:26.962606 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-75fe9f-openshift-default-7f6b4f67b7-kw7r8"] Jul 22 01:09:27.905569 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:27.905523 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" path="/var/lib/kubelet/pods/ec7ebb11-4e78-4c5a-812e-53e23404727c/volumes" Jul 22 01:09:30.291568 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.291527 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh"] Jul 22 01:09:30.292076 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.292055 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" Jul 22 01:09:30.292152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.292080 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" Jul 22 01:09:30.292152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.292103 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" Jul 22 01:09:30.292152 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.292111 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" Jul 22 01:09:30.292304 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.292214 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" Jul 22 01:09:30.292304 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.292231 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec7ebb11-4e78-4c5a-812e-53e23404727c" containerName="istio-proxy" Jul 22 01:09:30.296677 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.296635 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.299621 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.299600 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-956e89-openshift-default-dockercfg-dx7pq\"" Jul 22 01:09:30.316035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.316014 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh"] Jul 22 01:09:30.361606 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361572 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-certs\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.361775 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361623 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-credential-socket\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.361775 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361701 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-data\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.361775 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-socket\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.361920 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb83f3e-d34d-499c-9582-62aeb453cece-istiod-ca-cert\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.361920 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-token\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.361920 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-envoy\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.362039 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.361969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmwb\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-kube-api-access-rpmwb\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.362039 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.362006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-podinfo\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463057 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-envoy\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463255 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmwb\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-kube-api-access-rpmwb\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463255 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-podinfo\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463255 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-certs\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463255 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-credential-socket\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463255 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-data\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463255 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-socket\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463667 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb83f3e-d34d-499c-9582-62aeb453cece-istiod-ca-cert\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463667 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463335 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-token\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463667 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-certs\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463667 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-data\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463667 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-credential-socket\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463921 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463842 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-socket\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.463961 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.463936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb83f3e-d34d-499c-9582-62aeb453cece-istiod-ca-cert\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.465414 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.465397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-envoy\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.465512 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.465495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-podinfo\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.477160 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.477127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-token\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.477323 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.477305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmwb\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-kube-api-access-rpmwb\") pod \"e2e-api-b-956e89-openshift-default-6865467845-kk4xh\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.606597 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.606498 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:30.745551 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.745515 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh"] Jul 22 01:09:30.748872 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:09:30.748844 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb83f3e_d34d_499c_9582_62aeb453cece.slice/crio-ffeecd76e7d807838b13d431be37d0726986756476170efe06d1c90330a5a3e4 WatchSource:0}: Error finding container ffeecd76e7d807838b13d431be37d0726986756476170efe06d1c90330a5a3e4: Status 404 returned error can't find the container with id ffeecd76e7d807838b13d431be37d0726986756476170efe06d1c90330a5a3e4 Jul 22 01:09:30.751180 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.751149 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:30.751276 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.751214 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:30.751276 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.751241 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:30.946099 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.946008 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" event={"ID":"9eb83f3e-d34d-499c-9582-62aeb453cece","Type":"ContainerStarted","Data":"bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d"} Jul 22 01:09:30.946099 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.946059 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" event={"ID":"9eb83f3e-d34d-499c-9582-62aeb453cece","Type":"ContainerStarted","Data":"ffeecd76e7d807838b13d431be37d0726986756476170efe06d1c90330a5a3e4"} Jul 22 01:09:30.977387 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:30.977151 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podStartSLOduration=0.977129558 podStartE2EDuration="977.129558ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:09:30.975226338 +0000 UTC m=+2555.605458891" watchObservedRunningTime="2026-07-22 01:09:30.977129558 +0000 UTC m=+2555.607362100" Jul 22 01:09:31.606622 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:31.606589 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:31.611539 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:31.611513 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:31.950619 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:31.950530 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:31.951569 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:31.951547 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:32.292358 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:32.292328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:09:39.953802 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:39.953760 2562 patch_prober.go:28] interesting pod/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 01:09:39.954203 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:39.953851 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 01:09:41.001624 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:41.001594 2562 generic.go:356] "Generic (PLEG): container finished" podID="efaba292-633a-40d9-b1e5-219701710c06" containerID="1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989" exitCode=0 Jul 22 01:09:41.002031 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:41.001683 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" event={"ID":"efaba292-633a-40d9-b1e5-219701710c06","Type":"ContainerDied","Data":"1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989"} Jul 22 01:09:41.002098 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:41.002083 2562 scope.go:117] "RemoveContainer" containerID="1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989" Jul 22 01:09:41.003057 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:41.003026 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:41.003175 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:41.003110 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:41.003175 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:41.003157 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:42.009133 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:42.009099 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" event={"ID":"efaba292-633a-40d9-b1e5-219701710c06","Type":"ContainerStarted","Data":"29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2"} Jul 22 01:09:42.327325 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:42.327289 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:42.327559 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:42.327535 2562 patch_prober.go:28] interesting pod/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 22 01:09:42.327604 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:42.327584 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 22 01:09:43.328280 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:43.328246 2562 patch_prober.go:28] interesting pod/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 22 01:09:43.328705 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:43.328300 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 22 01:09:44.327818 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:44.327780 2562 patch_prober.go:28] interesting pod/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 22 01:09:44.328005 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:44.327840 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 22 01:09:45.328011 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:45.327978 2562 patch_prober.go:28] interesting pod/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 22 01:09:45.328456 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:45.328431 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 22 01:09:46.327399 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:46.327365 2562 patch_prober.go:28] interesting pod/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 22 01:09:46.327588 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:46.327438 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 22 01:09:47.328054 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:47.328007 2562 patch_prober.go:28] interesting pod/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 22 01:09:47.328529 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:47.328094 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 22 01:09:47.991841 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:47.991799 2562 patch_prober.go:28] interesting pod/e2e-api-b-956e89-openshift-default-6865467845-kk4xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 01:09:47.992020 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:47.991874 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 01:09:48.331387 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:48.331354 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:48.331788 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:48.331751 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:48.332432 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:48.332409 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:09:48.506729 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:48.506697 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:09:50.046109 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:50.046076 2562 generic.go:356] "Generic (PLEG): container finished" podID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerID="bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d" exitCode=0 Jul 22 01:09:50.046590 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:50.046147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" event={"ID":"9eb83f3e-d34d-499c-9582-62aeb453cece","Type":"ContainerDied","Data":"bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d"} Jul 22 01:09:50.046688 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:50.046609 2562 scope.go:117] "RemoveContainer" containerID="bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d" Jul 22 01:09:50.047527 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:50.047501 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:50.047586 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:50.047566 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:50.047621 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:50.047599 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:09:51.053075 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:51.053040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" event={"ID":"9eb83f3e-d34d-499c-9582-62aeb453cece","Type":"ContainerStarted","Data":"8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187"} Jul 22 01:09:51.607584 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:51.607545 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:51.607790 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:51.607763 2562 patch_prober.go:28] interesting pod/e2e-api-b-956e89-openshift-default-6865467845-kk4xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 01:09:51.607843 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:51.607810 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 01:09:52.607405 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:52.607372 2562 patch_prober.go:28] interesting pod/e2e-api-b-956e89-openshift-default-6865467845-kk4xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 01:09:52.607821 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:52.607439 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 01:09:53.607009 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:53.606976 2562 patch_prober.go:28] interesting pod/e2e-api-b-956e89-openshift-default-6865467845-kk4xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 01:09:53.607189 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:53.607044 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 01:09:54.607136 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:54.607094 2562 patch_prober.go:28] interesting pod/e2e-api-b-956e89-openshift-default-6865467845-kk4xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 22 01:09:54.607537 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:54.607157 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 22 01:09:55.610270 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:55.610240 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:55.610638 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:55.610613 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:09:55.611227 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:09:55.611209 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:10:04.782799 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:04.782759 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:10:12.622957 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:12.622927 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh"] Jul 22 01:10:12.623470 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:12.623113 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" containerID="cri-o://29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2" gracePeriod=30 Jul 22 01:10:17.875897 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:17.875868 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:10:18.024801 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.024720 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-envoy\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.024801 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.024756 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq4hw\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-kube-api-access-vq4hw\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.024801 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.024775 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaba292-633a-40d9-b1e5-219701710c06-istiod-ca-cert\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.024801 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.024802 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-data\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.025127 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.024949 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-certs\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.025127 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025010 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-istio-token\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.025127 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025040 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaba292-633a-40d9-b1e5-219701710c06-istio-podinfo\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.025127 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025102 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-socket\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.025328 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025178 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-credential-socket\") pod \"efaba292-633a-40d9-b1e5-219701710c06\" (UID: \"efaba292-633a-40d9-b1e5-219701710c06\") " Jul 22 01:10:18.025328 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025214 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:18.025328 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025246 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efaba292-633a-40d9-b1e5-219701710c06-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:10:18.025537 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025520 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efaba292-633a-40d9-b1e5-219701710c06-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.025601 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025542 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.025601 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025558 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:18.025601 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025588 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:18.025736 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.025674 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-data" (OuterVolumeSpecName: "istio-data") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:18.026996 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.026970 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-kube-api-access-vq4hw" (OuterVolumeSpecName: "kube-api-access-vq4hw") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "kube-api-access-vq4hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:10:18.027156 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.027138 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-istio-token" (OuterVolumeSpecName: "istio-token") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:10:18.027272 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.027253 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:18.027338 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.027326 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efaba292-633a-40d9-b1e5-219701710c06-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efaba292-633a-40d9-b1e5-219701710c06" (UID: "efaba292-633a-40d9-b1e5-219701710c06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:10:18.126490 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.126452 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.126490 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.126485 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.126490 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.126495 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4hw\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-kube-api-access-vq4hw\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.126725 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.126504 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.126725 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.126512 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efaba292-633a-40d9-b1e5-219701710c06-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.126725 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.126520 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efaba292-633a-40d9-b1e5-219701710c06-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.126725 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.126529 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efaba292-633a-40d9-b1e5-219701710c06-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:18.195337 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.195303 2562 generic.go:356] "Generic (PLEG): container finished" podID="efaba292-633a-40d9-b1e5-219701710c06" containerID="29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2" exitCode=0 Jul 22 01:10:18.195501 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.195357 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" Jul 22 01:10:18.195501 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.195373 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" event={"ID":"efaba292-633a-40d9-b1e5-219701710c06","Type":"ContainerDied","Data":"29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2"} Jul 22 01:10:18.195501 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.195415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh" event={"ID":"efaba292-633a-40d9-b1e5-219701710c06","Type":"ContainerDied","Data":"c2732eea1fd08bfd24162ba16490a3b723579771b68d65c78254ee50d9aa72d0"} Jul 22 01:10:18.195501 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.195438 2562 scope.go:117] "RemoveContainer" containerID="29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2" Jul 22 01:10:18.204851 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.204835 2562 scope.go:117] "RemoveContainer" containerID="1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989" Jul 22 01:10:18.212197 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.212179 2562 scope.go:117] "RemoveContainer" containerID="29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2" Jul 22 01:10:18.212430 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:10:18.212412 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2\": container with ID starting with 29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2 not found: ID does not exist" containerID="29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2" Jul 22 01:10:18.212486 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.212439 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2"} err="failed to get container status \"29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2\": rpc error: code = NotFound desc = could not find container \"29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2\": container with ID starting with 29a30ee9ab74384eae57653400e39f1b7b2d1031afc7c91d52d5e207364306e2 not found: ID does not exist" Jul 22 01:10:18.212486 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.212457 2562 scope.go:117] "RemoveContainer" containerID="1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989" Jul 22 01:10:18.212717 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:10:18.212690 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989\": container with ID starting with 1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989 not found: ID does not exist" containerID="1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989" Jul 22 01:10:18.212813 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.212723 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989"} err="failed to get container status \"1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989\": rpc error: code = NotFound desc = could not find container \"1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989\": container with ID starting with 1e17c1907cf4d08df0a426c67c01897e3e0933d037597e52806d2cd98e44c989 not found: ID does not exist" Jul 22 01:10:18.219379 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.219358 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh"] Jul 22 01:10:18.223404 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:18.223383 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-659cfa-openshift-default-6598cfcc47-mzzvh"] Jul 22 01:10:19.905781 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:19.905746 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efaba292-633a-40d9-b1e5-219701710c06" path="/var/lib/kubelet/pods/efaba292-633a-40d9-b1e5-219701710c06/volumes" Jul 22 01:10:32.025102 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:32.025069 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:10:39.581915 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:39.581883 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh"] Jul 22 01:10:39.582448 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:39.582088 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" containerID="cri-o://8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187" gracePeriod=30 Jul 22 01:10:40.455700 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.455631 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj"] Jul 22 01:10:40.456062 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.456050 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" Jul 22 01:10:40.456109 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.456064 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" Jul 22 01:10:40.456109 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.456077 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" Jul 22 01:10:40.456109 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.456082 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" Jul 22 01:10:40.456207 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.456139 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" Jul 22 01:10:40.456207 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.456147 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="efaba292-633a-40d9-b1e5-219701710c06" containerName="istio-proxy" Jul 22 01:10:40.458035 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.458018 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.460434 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.460413 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e4b9ac-openshift-default-dockercfg-sk5gw\"" Jul 22 01:10:40.472408 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.472384 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj"] Jul 22 01:10:40.516704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-data\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.516704 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qntm\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-kube-api-access-6qntm\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.516877 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-envoy\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.516877 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-istio-token\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.516877 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-certs\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.516877 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9542105e-98a9-4489-a181-852ee147c575-istiod-ca-cert\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.517007 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9542105e-98a9-4489-a181-852ee147c575-istio-podinfo\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.517007 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516907 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-socket\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.517007 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.516923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-credential-socket\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.612721 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.612680 2562 patch_prober.go:28] interesting pod/e2e-api-b-956e89-openshift-default-6865467845-kk4xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 01:10:40.613087 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.612747 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 01:10:40.618124 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-certs\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618179 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9542105e-98a9-4489-a181-852ee147c575-istiod-ca-cert\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618179 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618158 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9542105e-98a9-4489-a181-852ee147c575-istio-podinfo\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618281 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-socket\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618281 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-credential-socket\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618281 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-data\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qntm\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-kube-api-access-6qntm\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-envoy\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618425 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-istio-token\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618571 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-socket\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618771 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-credential-socket\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618914 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618857 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-data\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.618968 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.618901 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-certs\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.619034 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.619012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9542105e-98a9-4489-a181-852ee147c575-istiod-ca-cert\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.620578 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.620555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-envoy\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.620916 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.620897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9542105e-98a9-4489-a181-852ee147c575-istio-podinfo\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.627111 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.627086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-istio-token\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.627412 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.627389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qntm\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-kube-api-access-6qntm\") pod \"e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.767861 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.767779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:40.908112 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:10:40.908079 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9542105e_98a9_4489_a181_852ee147c575.slice/crio-5d1bb13c8b539eb08f99f2509cf6e0a1ec24a4ee30ca5e0557c3925ff9c91203 WatchSource:0}: Error finding container 5d1bb13c8b539eb08f99f2509cf6e0a1ec24a4ee30ca5e0557c3925ff9c91203: Status 404 returned error can't find the container with id 5d1bb13c8b539eb08f99f2509cf6e0a1ec24a4ee30ca5e0557c3925ff9c91203 Jul 22 01:10:40.910967 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.910924 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:10:40.911061 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.911022 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:10:40.911106 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.911063 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:10:40.911497 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:40.911477 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj"] Jul 22 01:10:41.306630 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:41.306590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" event={"ID":"9542105e-98a9-4489-a181-852ee147c575","Type":"ContainerStarted","Data":"73fe461c2e3b244358639e43507dfaa4a7fae515def26a105c69efbf2ee47efb"} Jul 22 01:10:41.306857 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:41.306674 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" event={"ID":"9542105e-98a9-4489-a181-852ee147c575","Type":"ContainerStarted","Data":"5d1bb13c8b539eb08f99f2509cf6e0a1ec24a4ee30ca5e0557c3925ff9c91203"} Jul 22 01:10:41.327545 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:41.327477 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" podStartSLOduration=1.3274588729999999 podStartE2EDuration="1.327458873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:10:41.326599875 +0000 UTC m=+2625.956832425" watchObservedRunningTime="2026-07-22 01:10:41.327458873 +0000 UTC m=+2625.957691415" Jul 22 01:10:41.768344 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:41.768264 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:41.773174 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:41.773152 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:42.311160 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:42.311131 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:42.312024 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:42.312003 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:10:44.851219 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.851155 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:10:44.953398 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953363 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-credential-socket\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953586 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953412 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb83f3e-d34d-499c-9582-62aeb453cece-istiod-ca-cert\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953586 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953460 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-data\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953586 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953491 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-envoy\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953780 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953687 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-socket\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953780 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-certs\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953780 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953742 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:44.953780 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953773 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-podinfo\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953995 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953831 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-token\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953995 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953891 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpmwb\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-kube-api-access-rpmwb\") pod \"9eb83f3e-d34d-499c-9582-62aeb453cece\" (UID: \"9eb83f3e-d34d-499c-9582-62aeb453cece\") " Jul 22 01:10:44.953995 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953934 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:44.953995 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953948 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:44.954203 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.953994 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb83f3e-d34d-499c-9582-62aeb453cece-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:10:44.954272 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.954215 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:44.954272 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.954234 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb83f3e-d34d-499c-9582-62aeb453cece-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:44.954272 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.954248 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:44.954272 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.954263 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:44.954465 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.954427 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-data" (OuterVolumeSpecName: "istio-data") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:44.955981 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.955954 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-token" (OuterVolumeSpecName: "istio-token") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:10:44.956092 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.956051 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-kube-api-access-rpmwb" (OuterVolumeSpecName: "kube-api-access-rpmwb") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "kube-api-access-rpmwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:10:44.956092 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.956064 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:10:44.956165 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:44.956118 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9eb83f3e-d34d-499c-9582-62aeb453cece" (UID: "9eb83f3e-d34d-499c-9582-62aeb453cece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:10:45.055305 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.055273 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:45.055305 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.055300 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:45.055305 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.055309 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmwb\" (UniqueName: \"kubernetes.io/projected/9eb83f3e-d34d-499c-9582-62aeb453cece-kube-api-access-rpmwb\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:45.055519 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.055318 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:45.055519 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.055327 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb83f3e-d34d-499c-9582-62aeb453cece-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:10:45.325436 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.325392 2562 generic.go:356] "Generic (PLEG): container finished" podID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerID="8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187" exitCode=0 Jul 22 01:10:45.325638 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.325451 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" Jul 22 01:10:45.325638 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.325485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" event={"ID":"9eb83f3e-d34d-499c-9582-62aeb453cece","Type":"ContainerDied","Data":"8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187"} Jul 22 01:10:45.325638 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.325549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh" event={"ID":"9eb83f3e-d34d-499c-9582-62aeb453cece","Type":"ContainerDied","Data":"ffeecd76e7d807838b13d431be37d0726986756476170efe06d1c90330a5a3e4"} Jul 22 01:10:45.325638 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.325583 2562 scope.go:117] "RemoveContainer" containerID="8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187" Jul 22 01:10:45.335820 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.335806 2562 scope.go:117] "RemoveContainer" containerID="bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d" Jul 22 01:10:45.343550 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.343533 2562 scope.go:117] "RemoveContainer" containerID="8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187" Jul 22 01:10:45.343838 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:10:45.343817 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187\": container with ID starting with 8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187 not found: ID does not exist" containerID="8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187" Jul 22 01:10:45.343893 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.343845 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187"} err="failed to get container status \"8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187\": rpc error: code = NotFound desc = could not find container \"8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187\": container with ID starting with 8d248273a3786c390dc5306580c92e44d1ff55cce27f433ad58dd7ffccf73187 not found: ID does not exist" Jul 22 01:10:45.343893 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.343863 2562 scope.go:117] "RemoveContainer" containerID="bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d" Jul 22 01:10:45.344078 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:10:45.344062 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d\": container with ID starting with bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d not found: ID does not exist" containerID="bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d" Jul 22 01:10:45.344116 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.344085 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d"} err="failed to get container status \"bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d\": rpc error: code = NotFound desc = could not find container \"bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d\": container with ID starting with bdf6064063ef0606a32212d726d63d08c895dd8a9a2da57a805be7724290815d not found: ID does not exist" Jul 22 01:10:45.348565 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.348548 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh"] Jul 22 01:10:45.353037 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.353016 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-956e89-openshift-default-6865467845-kk4xh"] Jul 22 01:10:45.905831 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:45.905796 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" path="/var/lib/kubelet/pods/9eb83f3e-d34d-499c-9582-62aeb453cece/volumes" Jul 22 01:10:48.374795 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.374758 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb"] Jul 22 01:10:48.375176 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.375110 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" Jul 22 01:10:48.375176 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.375121 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" Jul 22 01:10:48.375176 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.375131 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" Jul 22 01:10:48.375176 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.375137 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" Jul 22 01:10:48.375327 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.375188 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" Jul 22 01:10:48.375327 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.375197 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb83f3e-d34d-499c-9582-62aeb453cece" containerName="istio-proxy" Jul 22 01:10:48.378036 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.378020 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.380839 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.380813 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-4f35c1-openshift-default-dockercfg-prr8k\"" Jul 22 01:10:48.391104 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.391085 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb"] Jul 22 01:10:48.484910 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.484877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68qt\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-kube-api-access-n68qt\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485101 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.484924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-token\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485101 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.484943 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-socket\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485101 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.484969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-envoy\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485101 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.485002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-credential-socket\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485101 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.485022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-data\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485351 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.485106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-podinfo\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485351 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.485189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4717aff-537a-4c15-84fa-73dfd3239af6-istiod-ca-cert\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.485351 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.485244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-certs\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586606 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4717aff-537a-4c15-84fa-73dfd3239af6-istiod-ca-cert\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586810 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-certs\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586810 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68qt\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-kube-api-access-n68qt\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586810 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586721 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-token\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586975 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-socket\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586975 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-envoy\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586975 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-credential-socket\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.586975 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.586948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-data\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.587198 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.587000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-podinfo\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.587198 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.587114 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-certs\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.587311 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.587283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-credential-socket\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.587382 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.587354 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4717aff-537a-4c15-84fa-73dfd3239af6-istiod-ca-cert\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.587382 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.587370 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-socket\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.587487 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.587428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-data\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.589617 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.589591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-envoy\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.589788 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.589695 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-podinfo\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.595051 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.595022 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-token\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.595334 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.595312 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68qt\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-kube-api-access-n68qt\") pod \"e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.688437 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.688355 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:48.819218 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.819191 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb"] Jul 22 01:10:48.821848 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:10:48.821811 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4717aff_537a_4c15_84fa_73dfd3239af6.slice/crio-36d3411ae3e58d95aa67e372bfc6b89f0a481ff2841eae5ee8538d307a1b3a3b WatchSource:0}: Error finding container 36d3411ae3e58d95aa67e372bfc6b89f0a481ff2841eae5ee8538d307a1b3a3b: Status 404 returned error can't find the container with id 36d3411ae3e58d95aa67e372bfc6b89f0a481ff2841eae5ee8538d307a1b3a3b Jul 22 01:10:48.823890 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.823857 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:10:48.823985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.823939 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:10:48.824048 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:48.823985 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:10:49.349097 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:49.349065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" event={"ID":"a4717aff-537a-4c15-84fa-73dfd3239af6","Type":"ContainerStarted","Data":"f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb"} Jul 22 01:10:49.349267 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:49.349114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" event={"ID":"a4717aff-537a-4c15-84fa-73dfd3239af6","Type":"ContainerStarted","Data":"36d3411ae3e58d95aa67e372bfc6b89f0a481ff2841eae5ee8538d307a1b3a3b"} Jul 22 01:10:49.370087 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:49.370039 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" podStartSLOduration=1.370025405 podStartE2EDuration="1.370025405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:10:49.36912837 +0000 UTC m=+2633.999360936" watchObservedRunningTime="2026-07-22 01:10:49.370025405 +0000 UTC m=+2634.000257945" Jul 22 01:10:49.689262 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:49.689182 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:49.694052 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:49.694028 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:49.884770 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:49.884739 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:10:50.354402 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:50.354375 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:50.355345 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:50.355322 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:10:58.352865 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:58.352801 2562 patch_prober.go:28] interesting pod/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 01:10:58.353807 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:10:58.353482 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 01:11:00.411195 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:00.411162 2562 generic.go:356] "Generic (PLEG): container finished" podID="9542105e-98a9-4489-a181-852ee147c575" containerID="73fe461c2e3b244358639e43507dfaa4a7fae515def26a105c69efbf2ee47efb" exitCode=0 Jul 22 01:11:00.411722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:00.411203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" event={"ID":"9542105e-98a9-4489-a181-852ee147c575","Type":"ContainerDied","Data":"73fe461c2e3b244358639e43507dfaa4a7fae515def26a105c69efbf2ee47efb"} Jul 22 01:11:00.411722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:00.411702 2562 scope.go:117] "RemoveContainer" containerID="73fe461c2e3b244358639e43507dfaa4a7fae515def26a105c69efbf2ee47efb" Jul 22 01:11:00.412722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:00.412692 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:11:00.412834 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:00.412759 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:11:00.412834 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:00.412794 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:11:01.418363 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:01.418329 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" event={"ID":"9542105e-98a9-4489-a181-852ee147c575","Type":"ContainerStarted","Data":"c4167107c0755e2f695e88aff335cc17481dfd6829b39ec0e02445b2a06799b2"} Jul 22 01:11:01.768940 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:01.768845 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:11:01.769591 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:01.769565 2562 patch_prober.go:28] interesting pod/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 22 01:11:01.769851 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:01.769809 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 22 01:11:01.792307 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:01.792277 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:11:02.769118 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:02.769078 2562 patch_prober.go:28] interesting pod/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 22 01:11:02.769619 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:02.769152 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 22 01:11:03.428719 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.428688 2562 generic.go:356] "Generic (PLEG): container finished" podID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerID="f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb" exitCode=0 Jul 22 01:11:03.428972 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.428763 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" event={"ID":"a4717aff-537a-4c15-84fa-73dfd3239af6","Type":"ContainerDied","Data":"f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb"} Jul 22 01:11:03.429251 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.429234 2562 scope.go:117] "RemoveContainer" containerID="f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb" Jul 22 01:11:03.430257 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.430229 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:11:03.430353 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.430296 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:11:03.430353 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.430327 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 22 01:11:03.768685 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.768571 2562 patch_prober.go:28] interesting pod/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 22 01:11:03.768685 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:03.768659 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 22 01:11:04.435804 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:04.435777 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" event={"ID":"a4717aff-537a-4c15-84fa-73dfd3239af6","Type":"ContainerStarted","Data":"c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd"} Jul 22 01:11:04.690528 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:04.690420 2562 patch_prober.go:28] interesting pod/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 01:11:04.690528 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:04.690492 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 01:11:04.690799 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:04.690786 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:11:04.768589 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:04.768544 2562 patch_prober.go:28] interesting pod/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 22 01:11:04.768817 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:04.768629 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 22 01:11:05.354691 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:05.354634 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:11:05.688949 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:05.688865 2562 patch_prober.go:28] interesting pod/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 01:11:05.688949 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:05.688916 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 01:11:05.774036 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:05.774004 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:11:05.774295 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:05.774276 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:11:05.774956 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:05.774935 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:11:06.689319 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:06.689284 2562 patch_prober.go:28] interesting pod/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 01:11:06.689767 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:06.689340 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 01:11:07.689050 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:07.689013 2562 patch_prober.go:28] interesting pod/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 22 01:11:07.689219 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:07.689069 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 22 01:11:08.692530 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:08.692498 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:11:08.693563 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:08.693547 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:11:12.993660 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:12.993610 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:11:56.005745 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:56.005672 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:11:56.011366 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:11:56.011340 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:12:02.187531 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:12:02.187463 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:12:12.582084 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:12:12.582046 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:12:27.681101 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:12:27.681059 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:12:46.072868 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:12:46.072831 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:13:03.382145 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:13:03.382114 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:13:19.281013 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:13:19.280933 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:13:37.781445 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:13:37.781402 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:13:55.784124 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:13:55.784089 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:14:20.682203 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:14:20.682173 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:14:48.783607 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:14:48.783533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:15:09.380285 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:09.380252 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:15:27.385180 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:27.385149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:15:37.514357 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.514324 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc"] Jul 22 01:15:37.517600 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.517584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.520157 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.520136 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"kube-root-ca.crt\"" Jul 22 01:15:37.520253 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.520183 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"openshift-service-ca.crt\"" Jul 22 01:15:37.520840 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.520823 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"default-dockercfg-c4cv2\"" Jul 22 01:15:37.520840 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.520830 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e4b9ac\"/\"rate-test-model-e4b9ac-kserve-self-signed-certs\"" Jul 22 01:15:37.526857 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.526835 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc"] Jul 22 01:15:37.598534 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.598509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tmp-dir\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.598622 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.598556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjz8\" (UniqueName: \"kubernetes.io/projected/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-kube-api-access-swjz8\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.598622 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.598585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-model-cache\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.598622 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.598613 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-dshm\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.598753 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.598686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-home\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.598753 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.598704 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tls-certs\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.700207 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.700178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tmp-dir\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.700482 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.700456 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjz8\" (UniqueName: \"kubernetes.io/projected/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-kube-api-access-swjz8\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.700608 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.700496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-model-cache\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.700608 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.700536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-dshm\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.700753 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.700607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-home\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.700753 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.700620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tmp-dir\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.700753 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.700634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tls-certs\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.701030 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.701010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-home\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.701174 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.701076 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-model-cache\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.703163 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.703141 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-dshm\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.703761 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.703739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tls-certs\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.713383 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.713350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjz8\" (UniqueName: \"kubernetes.io/projected/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-kube-api-access-swjz8\") pod \"rate-test-model-e4b9ac-kserve-56d998785f-wxpkc\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.826741 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.826717 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:37.957826 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:37.957787 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc"] Jul 22 01:15:37.960378 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:15:37.960351 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4469b8af_969b_4f4c_b5b5_18aa2cd1360a.slice/crio-0e0e507c89efd4e1411d7ec81d0fcfbc0a9f134e48528dc158db77e8fc5639cd WatchSource:0}: Error finding container 0e0e507c89efd4e1411d7ec81d0fcfbc0a9f134e48528dc158db77e8fc5639cd: Status 404 returned error can't find the container with id 0e0e507c89efd4e1411d7ec81d0fcfbc0a9f134e48528dc158db77e8fc5639cd Jul 22 01:15:38.650531 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:38.650505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" event={"ID":"4469b8af-969b-4f4c-b5b5-18aa2cd1360a","Type":"ContainerStarted","Data":"4f7d6fd3f6d1f69c75fcdc7070a48ddc0eea6d43334516a0f505e4f74027bc2e"} Jul 22 01:15:38.650907 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:38.650539 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" event={"ID":"4469b8af-969b-4f4c-b5b5-18aa2cd1360a","Type":"ContainerStarted","Data":"0e0e507c89efd4e1411d7ec81d0fcfbc0a9f134e48528dc158db77e8fc5639cd"} Jul 22 01:15:38.668989 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:38.668945 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" podStartSLOduration=1.668930714 podStartE2EDuration="1.668930714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:15:38.667958436 +0000 UTC m=+2923.298190988" watchObservedRunningTime="2026-07-22 01:15:38.668930714 +0000 UTC m=+2923.299163253" Jul 22 01:15:46.086383 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:46.086351 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:15:47.827492 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:47.827467 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:47.827940 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:47.827504 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:47.839884 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:47.839861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:15:48.710021 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:15:48.709991 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:16:11.279379 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:16:11.279343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:16:27.287245 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:16:27.287172 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:16:49.212701 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:16:49.212659 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:16:53.841538 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:16:53.841494 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc"] Jul 22 01:16:56.033482 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:16:56.033446 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:16:56.040432 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:16:56.040409 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zsrgc_2f487ea8-8634-47a0-afc1-22b60a19e52a/ovn-acl-logging/0.log" Jul 22 01:17:04.177029 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:04.176995 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:17:08.861841 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:08.861769 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" podUID="4469b8af-969b-4f4c-b5b5-18aa2cd1360a" containerName="main" containerID="cri-o://4f7d6fd3f6d1f69c75fcdc7070a48ddc0eea6d43334516a0f505e4f74027bc2e" gracePeriod=45 Jul 22 01:17:09.064471 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.064442 2562 generic.go:356] "Generic (PLEG): container finished" podID="4469b8af-969b-4f4c-b5b5-18aa2cd1360a" containerID="4f7d6fd3f6d1f69c75fcdc7070a48ddc0eea6d43334516a0f505e4f74027bc2e" exitCode=0 Jul 22 01:17:09.064611 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.064523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" event={"ID":"4469b8af-969b-4f4c-b5b5-18aa2cd1360a","Type":"ContainerDied","Data":"4f7d6fd3f6d1f69c75fcdc7070a48ddc0eea6d43334516a0f505e4f74027bc2e"} Jul 22 01:17:09.107607 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.107584 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:17:09.149620 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.149553 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swjz8\" (UniqueName: \"kubernetes.io/projected/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-kube-api-access-swjz8\") pod \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " Jul 22 01:17:09.149620 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.149605 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tls-certs\") pod \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " Jul 22 01:17:09.149820 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.149682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tmp-dir\") pod \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " Jul 22 01:17:09.149820 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.149709 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-dshm\") pod \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " Jul 22 01:17:09.149820 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.149738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-home\") pod \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " Jul 22 01:17:09.149968 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.149825 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-model-cache\") pod \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\" (UID: \"4469b8af-969b-4f4c-b5b5-18aa2cd1360a\") " Jul 22 01:17:09.149968 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.149907 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4469b8af-969b-4f4c-b5b5-18aa2cd1360a" (UID: "4469b8af-969b-4f4c-b5b5-18aa2cd1360a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:09.150077 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.150032 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-home" (OuterVolumeSpecName: "home") pod "4469b8af-969b-4f4c-b5b5-18aa2cd1360a" (UID: "4469b8af-969b-4f4c-b5b5-18aa2cd1360a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:09.150150 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.150128 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4469b8af-969b-4f4c-b5b5-18aa2cd1360a" (UID: "4469b8af-969b-4f4c-b5b5-18aa2cd1360a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:09.150232 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.150218 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-model-cache\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:09.150294 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.150238 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tmp-dir\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:09.150294 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.150251 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-home\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:09.151887 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.151857 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4469b8af-969b-4f4c-b5b5-18aa2cd1360a" (UID: "4469b8af-969b-4f4c-b5b5-18aa2cd1360a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:17:09.151979 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.151907 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-kube-api-access-swjz8" (OuterVolumeSpecName: "kube-api-access-swjz8") pod "4469b8af-969b-4f4c-b5b5-18aa2cd1360a" (UID: "4469b8af-969b-4f4c-b5b5-18aa2cd1360a"). InnerVolumeSpecName "kube-api-access-swjz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:09.151979 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.151905 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-dshm" (OuterVolumeSpecName: "dshm") pod "4469b8af-969b-4f4c-b5b5-18aa2cd1360a" (UID: "4469b8af-969b-4f4c-b5b5-18aa2cd1360a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:09.251027 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.250988 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjz8\" (UniqueName: \"kubernetes.io/projected/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-kube-api-access-swjz8\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:09.251027 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.251024 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-tls-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:09.251188 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:09.251039 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4469b8af-969b-4f4c-b5b5-18aa2cd1360a-dshm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:10.069980 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:10.069948 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" Jul 22 01:17:10.070475 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:10.069947 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc" event={"ID":"4469b8af-969b-4f4c-b5b5-18aa2cd1360a","Type":"ContainerDied","Data":"0e0e507c89efd4e1411d7ec81d0fcfbc0a9f134e48528dc158db77e8fc5639cd"} Jul 22 01:17:10.070475 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:10.070085 2562 scope.go:117] "RemoveContainer" containerID="4f7d6fd3f6d1f69c75fcdc7070a48ddc0eea6d43334516a0f505e4f74027bc2e" Jul 22 01:17:10.087277 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:10.087256 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc"] Jul 22 01:17:10.090968 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:10.090946 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e4b9ac/rate-test-model-e4b9ac-kserve-56d998785f-wxpkc"] Jul 22 01:17:11.905649 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:11.905610 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4469b8af-969b-4f4c-b5b5-18aa2cd1360a" path="/var/lib/kubelet/pods/4469b8af-969b-4f4c-b5b5-18aa2cd1360a/volumes" Jul 22 01:17:27.275191 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:27.275160 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:17:36.018488 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:36.018455 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj"] Jul 22 01:17:36.019036 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:36.018698 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" containerID="cri-o://c4167107c0755e2f695e88aff335cc17481dfd6829b39ec0e02445b2a06799b2" gracePeriod=30 Jul 22 01:17:41.213067 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.213037 2562 generic.go:356] "Generic (PLEG): container finished" podID="9542105e-98a9-4489-a181-852ee147c575" containerID="c4167107c0755e2f695e88aff335cc17481dfd6829b39ec0e02445b2a06799b2" exitCode=0 Jul 22 01:17:41.213452 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.213090 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" event={"ID":"9542105e-98a9-4489-a181-852ee147c575","Type":"ContainerDied","Data":"c4167107c0755e2f695e88aff335cc17481dfd6829b39ec0e02445b2a06799b2"} Jul 22 01:17:41.213452 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.213151 2562 scope.go:117] "RemoveContainer" containerID="73fe461c2e3b244358639e43507dfaa4a7fae515def26a105c69efbf2ee47efb" Jul 22 01:17:41.269268 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.269245 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:17:41.421569 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421542 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-data\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.421722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421591 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qntm\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-kube-api-access-6qntm\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.421722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421611 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-envoy\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.421722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421635 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9542105e-98a9-4489-a181-852ee147c575-istiod-ca-cert\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.421722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421671 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-socket\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.421722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421687 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-istio-token\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.421722 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421706 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-certs\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.422039 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421735 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9542105e-98a9-4489-a181-852ee147c575-istio-podinfo\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.422039 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421797 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-credential-socket\") pod \"9542105e-98a9-4489-a181-852ee147c575\" (UID: \"9542105e-98a9-4489-a181-852ee147c575\") " Jul 22 01:17:41.422039 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.421991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:41.422209 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.422073 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9542105e-98a9-4489-a181-852ee147c575-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:17:41.422209 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.422086 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.422312 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.422266 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:41.422369 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.422330 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:41.422604 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.422578 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-data" (OuterVolumeSpecName: "istio-data") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:41.423783 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.423758 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:17:41.423783 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.423774 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-istio-token" (OuterVolumeSpecName: "istio-token") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:41.423941 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.423878 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-kube-api-access-6qntm" (OuterVolumeSpecName: "kube-api-access-6qntm") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "kube-api-access-6qntm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:41.423986 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.423955 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9542105e-98a9-4489-a181-852ee147c575-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9542105e-98a9-4489-a181-852ee147c575" (UID: "9542105e-98a9-4489-a181-852ee147c575"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:17:41.523229 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523205 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qntm\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-kube-api-access-6qntm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.523229 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523226 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.523346 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523235 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9542105e-98a9-4489-a181-852ee147c575-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.523346 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523244 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9542105e-98a9-4489-a181-852ee147c575-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.523346 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523251 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.523346 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523259 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9542105e-98a9-4489-a181-852ee147c575-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.523346 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523267 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:41.523346 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:41.523274 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9542105e-98a9-4489-a181-852ee147c575-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:17:42.218688 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:42.218611 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" Jul 22 01:17:42.218688 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:42.218629 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj" event={"ID":"9542105e-98a9-4489-a181-852ee147c575","Type":"ContainerDied","Data":"5d1bb13c8b539eb08f99f2509cf6e0a1ec24a4ee30ca5e0557c3925ff9c91203"} Jul 22 01:17:42.219074 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:42.218699 2562 scope.go:117] "RemoveContainer" containerID="c4167107c0755e2f695e88aff335cc17481dfd6829b39ec0e02445b2a06799b2" Jul 22 01:17:42.234322 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:42.234299 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj"] Jul 22 01:17:42.237830 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:42.237808 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e4b9ac-openshift-default-599fd65b55-mh7rj"] Jul 22 01:17:43.905003 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:43.904970 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9542105e-98a9-4489-a181-852ee147c575" path="/var/lib/kubelet/pods/9542105e-98a9-4489-a181-852ee147c575/volumes" Jul 22 01:17:49.690833 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:49.690756 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vv8vn"] Jul 22 01:17:58.032743 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:58.032712 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb"] Jul 22 01:17:58.033193 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:17:58.032909 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" containerID="cri-o://c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd" gracePeriod=30 Jul 22 01:18:03.282293 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.282271 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:18:03.313000 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.312922 2562 generic.go:356] "Generic (PLEG): container finished" podID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerID="c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd" exitCode=0 Jul 22 01:18:03.313000 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.312969 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" event={"ID":"a4717aff-537a-4c15-84fa-73dfd3239af6","Type":"ContainerDied","Data":"c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd"} Jul 22 01:18:03.313000 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.312975 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" Jul 22 01:18:03.313276 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.313022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb" event={"ID":"a4717aff-537a-4c15-84fa-73dfd3239af6","Type":"ContainerDied","Data":"36d3411ae3e58d95aa67e372bfc6b89f0a481ff2841eae5ee8538d307a1b3a3b"} Jul 22 01:18:03.313276 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.313051 2562 scope.go:117] "RemoveContainer" containerID="c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd" Jul 22 01:18:03.322993 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.322969 2562 scope.go:117] "RemoveContainer" containerID="f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb" Jul 22 01:18:03.335501 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.335479 2562 scope.go:117] "RemoveContainer" containerID="c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd" Jul 22 01:18:03.335802 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:18:03.335783 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd\": container with ID starting with c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd not found: ID does not exist" containerID="c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd" Jul 22 01:18:03.335862 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.335808 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd"} err="failed to get container status \"c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd\": rpc error: code = NotFound desc = could not find container \"c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd\": container with ID starting with c7392934f0e40fa93f05f06faec5057701fd6b6b832a088577004d95036fc4fd not found: ID does not exist" Jul 22 01:18:03.335862 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.335826 2562 scope.go:117] "RemoveContainer" containerID="f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb" Jul 22 01:18:03.336081 ip-10-0-131-63 kubenswrapper[2562]: E0722 01:18:03.336062 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb\": container with ID starting with f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb not found: ID does not exist" containerID="f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb" Jul 22 01:18:03.336139 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.336088 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb"} err="failed to get container status \"f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb\": rpc error: code = NotFound desc = could not find container \"f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb\": container with ID starting with f958a3321587ffb027aa41bb2ba559054069e3eff3d52648fd62674ac66d1bbb not found: ID does not exist" Jul 22 01:18:03.393514 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393491 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n68qt\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-kube-api-access-n68qt\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393594 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393535 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-token\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393594 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393570 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-credential-socket\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393705 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393606 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4717aff-537a-4c15-84fa-73dfd3239af6-istiod-ca-cert\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393705 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393628 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-envoy\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393705 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393668 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-podinfo\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393705 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393692 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-socket\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393910 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393725 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-certs\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.393910 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393759 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-data\") pod \"a4717aff-537a-4c15-84fa-73dfd3239af6\" (UID: \"a4717aff-537a-4c15-84fa-73dfd3239af6\") " Jul 22 01:18:03.394017 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.393961 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:18:03.394087 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.394051 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.394087 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.394058 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4717aff-537a-4c15-84fa-73dfd3239af6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:18:03.394195 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.394148 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:18:03.394353 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.394312 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:18:03.395027 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.395002 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:18:03.395728 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.395703 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:18:03.395821 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.395794 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:18:03.395870 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.395812 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 01:18:03.395870 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.395849 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-kube-api-access-n68qt" (OuterVolumeSpecName: "kube-api-access-n68qt") pod "a4717aff-537a-4c15-84fa-73dfd3239af6" (UID: "a4717aff-537a-4c15-84fa-73dfd3239af6"). InnerVolumeSpecName "kube-api-access-n68qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:18:03.494944 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494920 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.494944 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494943 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4717aff-537a-4c15-84fa-73dfd3239af6-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.495066 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494952 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.495066 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494960 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.495066 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494968 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.495066 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494975 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.495066 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494982 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68qt\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-kube-api-access-n68qt\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.495066 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.494990 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4717aff-537a-4c15-84fa-73dfd3239af6-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 22 01:18:03.635902 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.635875 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb"] Jul 22 01:18:03.638868 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.638849 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-4f35c1-openshift-default-5f5d49b879-nm5lb"] Jul 22 01:18:03.905799 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:03.905735 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" path="/var/lib/kubelet/pods/a4717aff-537a-4c15-84fa-73dfd3239af6/volumes" Jul 22 01:18:04.310898 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:04.310879 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rg8qz_6ddcc417-9097-475a-924a-456207977712/manager/0.log" Jul 22 01:18:04.428712 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:04.428693 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8w8ss_776cedef-60e1-4cae-a066-545d90fd56a8/manager/0.log" Jul 22 01:18:04.943537 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:04.943509 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-j7rgz_f94935d9-4b4b-41df-934f-4208f1ad6386/copy-manifests/0.log" Jul 22 01:18:04.952374 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:04.952353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-j7rgz_f94935d9-4b4b-41df-934f-4208f1ad6386/manager/0.log" Jul 22 01:18:07.512942 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:07.512911 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-52w5l_e59ba304-594e-45a4-b49f-5c1929122242/manager/0.log" Jul 22 01:18:07.635895 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:07.635865 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xxsmf_7c568dff-b531-417d-8393-8abf64065029/manager/0.log" Jul 22 01:18:08.107842 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:08.107816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vv8vn_e2df5edb-55ca-4a21-b04c-1b714c2564ce/limitador/0.log" Jul 22 01:18:08.569320 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:08.569292 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hgmfl_174e6bc7-959e-4306-ba1e-7d5d06bdb080/istio-proxy/0.log" Jul 22 01:18:09.038112 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:09.038018 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pqbfw_60abed7c-dc00-409b-9613-99580bb71748/istio-proxy/1.log" Jul 22 01:18:10.386053 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:10.386026 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dt6cv_5038e6ad-bd08-4ef8-84de-946d64735e72/main/0.log" Jul 22 01:18:18.948080 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:18.948036 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dzkpd_711d23ce-641f-4bf9-bf7a-cce1d84a05b5/global-pull-secret-syncer/0.log" Jul 22 01:18:19.158405 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:19.158378 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x9ndd_e48b5beb-479e-402e-891a-33a0c4c0449c/konnectivity-agent/0.log" Jul 22 01:18:19.220791 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:19.220719 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-63.ec2.internal_cee79eddbd8dbbcb832bb8dad2fba384/haproxy/0.log" Jul 22 01:18:25.047420 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:25.047390 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-52w5l_e59ba304-594e-45a4-b49f-5c1929122242/manager/0.log" Jul 22 01:18:25.080036 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:25.080008 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xxsmf_7c568dff-b531-417d-8393-8abf64065029/manager/0.log" Jul 22 01:18:25.325110 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:25.325028 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vv8vn_e2df5edb-55ca-4a21-b04c-1b714c2564ce/limitador/0.log" Jul 22 01:18:27.372721 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.372693 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69c588d6d6-kspc5_5d8a5554-103d-4375-81cb-003322c5d5fb/metrics-server/0.log" Jul 22 01:18:27.404636 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.404561 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-45wlh_64a31bc3-e4e2-47ef-a515-c1bef899f354/monitoring-plugin/0.log" Jul 22 01:18:27.434050 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.434023 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frwrs_f308f250-c886-4296-8b2a-e8e820b1dc6c/node-exporter/0.log" Jul 22 01:18:27.459841 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.459816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frwrs_f308f250-c886-4296-8b2a-e8e820b1dc6c/kube-rbac-proxy/0.log" Jul 22 01:18:27.482915 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.482894 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frwrs_f308f250-c886-4296-8b2a-e8e820b1dc6c/init-textfile/0.log" Jul 22 01:18:27.778814 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.778736 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6af90903-42d2-4580-aa55-cd028372c053/prometheus/0.log" Jul 22 01:18:27.807685 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.807638 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6af90903-42d2-4580-aa55-cd028372c053/config-reloader/0.log" Jul 22 01:18:27.834907 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.834874 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6af90903-42d2-4580-aa55-cd028372c053/thanos-sidecar/0.log" Jul 22 01:18:27.857500 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.857478 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6af90903-42d2-4580-aa55-cd028372c053/kube-rbac-proxy-web/0.log" Jul 22 01:18:27.881302 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.881268 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6af90903-42d2-4580-aa55-cd028372c053/kube-rbac-proxy/0.log" Jul 22 01:18:27.909669 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.909616 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6af90903-42d2-4580-aa55-cd028372c053/kube-rbac-proxy-thanos/0.log" Jul 22 01:18:27.933467 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:27.933451 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6af90903-42d2-4580-aa55-cd028372c053/init-config-reloader/0.log" Jul 22 01:18:28.022143 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:28.022116 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-4m9j6_bfbbacb0-1be5-40ea-be1d-829bb300f392/prometheus-operator-admission-webhook/0.log" Jul 22 01:18:28.143459 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:28.143434 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f766cb47-dwg8j_411b43ef-a9a2-4221-8057-db8435d41fea/thanos-query/0.log" Jul 22 01:18:28.165705 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:28.165680 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f766cb47-dwg8j_411b43ef-a9a2-4221-8057-db8435d41fea/kube-rbac-proxy-web/0.log" Jul 22 01:18:28.189913 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:28.189885 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f766cb47-dwg8j_411b43ef-a9a2-4221-8057-db8435d41fea/kube-rbac-proxy/0.log" Jul 22 01:18:28.212329 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:28.212305 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f766cb47-dwg8j_411b43ef-a9a2-4221-8057-db8435d41fea/prom-label-proxy/0.log" Jul 22 01:18:28.233256 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:28.233232 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f766cb47-dwg8j_411b43ef-a9a2-4221-8057-db8435d41fea/kube-rbac-proxy-rules/0.log" Jul 22 01:18:28.254234 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:28.254214 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67f766cb47-dwg8j_411b43ef-a9a2-4221-8057-db8435d41fea/kube-rbac-proxy-metrics/0.log" Jul 22 01:18:29.393007 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.392972 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54"] Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393331 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4469b8af-969b-4f4c-b5b5-18aa2cd1360a" containerName="main" Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393342 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4469b8af-969b-4f4c-b5b5-18aa2cd1360a" containerName="main" Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393351 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393356 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393364 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393369 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393375 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" Jul 22 01:18:29.393386 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393381 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" Jul 22 01:18:29.393665 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393394 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" Jul 22 01:18:29.393665 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393400 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" Jul 22 01:18:29.393665 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393466 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4469b8af-969b-4f4c-b5b5-18aa2cd1360a" containerName="main" Jul 22 01:18:29.393665 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393475 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" Jul 22 01:18:29.393665 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393483 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9542105e-98a9-4489-a181-852ee147c575" containerName="istio-proxy" Jul 22 01:18:29.393665 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393490 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" Jul 22 01:18:29.393665 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.393496 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4717aff-537a-4c15-84fa-73dfd3239af6" containerName="istio-proxy" Jul 22 01:18:29.396689 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.396668 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.399093 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.399070 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfz9r\"/\"openshift-service-ca.crt\"" Jul 22 01:18:29.399224 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.399070 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sfz9r\"/\"default-dockercfg-8wxxf\"" Jul 22 01:18:29.399224 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.399140 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfz9r\"/\"kube-root-ca.crt\"" Jul 22 01:18:29.407527 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.407503 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54"] Jul 22 01:18:29.518988 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.518954 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-lib-modules\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.519173 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.518997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-sys\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.519173 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.519032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzdl\" (UniqueName: \"kubernetes.io/projected/2ff77344-612d-4b47-af10-414cbc957382-kube-api-access-xjzdl\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.519173 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.519131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-proc\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.519299 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.519176 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-podres\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.619670 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-proc\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.619871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-podres\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.619871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-lib-modules\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.619871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-sys\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.619871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzdl\" (UniqueName: \"kubernetes.io/projected/2ff77344-612d-4b47-af10-414cbc957382-kube-api-access-xjzdl\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.619871 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-proc\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.620074 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619870 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-lib-modules\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.620074 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-sys\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.620074 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.619873 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ff77344-612d-4b47-af10-414cbc957382-podres\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.627983 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.627961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzdl\" (UniqueName: \"kubernetes.io/projected/2ff77344-612d-4b47-af10-414cbc957382-kube-api-access-xjzdl\") pod \"perf-node-gather-daemonset-twm54\" (UID: \"2ff77344-612d-4b47-af10-414cbc957382\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.706363 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.706268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:29.841989 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:29.841961 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54"] Jul 22 01:18:29.844363 ip-10-0-131-63 kubenswrapper[2562]: W0722 01:18:29.844334 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2ff77344_612d_4b47_af10_414cbc957382.slice/crio-a35d0351b6687e2ca3cce9f7299fbe32157a769da770e1d73586e3deec92878a WatchSource:0}: Error finding container a35d0351b6687e2ca3cce9f7299fbe32157a769da770e1d73586e3deec92878a: Status 404 returned error can't find the container with id a35d0351b6687e2ca3cce9f7299fbe32157a769da770e1d73586e3deec92878a Jul 22 01:18:30.437173 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:30.437132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" event={"ID":"2ff77344-612d-4b47-af10-414cbc957382","Type":"ContainerStarted","Data":"7843808158f70df5ed2538455743cf6f1425cfe3b5ddf7741815d4e2a763ecc6"} Jul 22 01:18:30.437590 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:30.437187 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" event={"ID":"2ff77344-612d-4b47-af10-414cbc957382","Type":"ContainerStarted","Data":"a35d0351b6687e2ca3cce9f7299fbe32157a769da770e1d73586e3deec92878a"} Jul 22 01:18:30.437590 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:30.437318 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:30.455877 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:30.455822 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" podStartSLOduration=1.455807179 podStartE2EDuration="1.455807179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:18:30.455000719 +0000 UTC m=+3095.085233261" watchObservedRunningTime="2026-07-22 01:18:30.455807179 +0000 UTC m=+3095.086039720" Jul 22 01:18:32.229985 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:32.229955 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44b7j_c995ce90-2e54-4a9e-b74a-e579d7643edc/dns/0.log" Jul 22 01:18:32.252909 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:32.252879 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-44b7j_c995ce90-2e54-4a9e-b74a-e579d7643edc/kube-rbac-proxy/0.log" Jul 22 01:18:32.379251 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:32.379219 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jdphr_441030f4-d8a5-48cf-82d4-4882497d5560/dns-node-resolver/0.log" Jul 22 01:18:33.082047 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:33.082014 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkzlt_71460dc1-1655-4123-b9d8-ded03647803a/node-ca/0.log" Jul 22 01:18:34.060773 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:34.060747 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hgmfl_174e6bc7-959e-4306-ba1e-7d5d06bdb080/istio-proxy/0.log" Jul 22 01:18:34.273520 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:34.273486 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pqbfw_60abed7c-dc00-409b-9613-99580bb71748/istio-proxy/0.log" Jul 22 01:18:34.301410 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:34.301376 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pqbfw_60abed7c-dc00-409b-9613-99580bb71748/istio-proxy/1.log" Jul 22 01:18:35.142606 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:35.142572 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zfwgj_0cba53b4-96f3-4b32-8f07-9f287fbae77e/serve-healthcheck-canary/0.log" Jul 22 01:18:35.963942 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:35.963912 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpmxl_af504347-b13b-40bd-89ed-3625443321ae/kube-rbac-proxy/0.log" Jul 22 01:18:35.991363 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:35.991331 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpmxl_af504347-b13b-40bd-89ed-3625443321ae/exporter/0.log" Jul 22 01:18:36.024313 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:36.024284 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpmxl_af504347-b13b-40bd-89ed-3625443321ae/extractor/0.log" Jul 22 01:18:36.451888 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:36.451858 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-twm54" Jul 22 01:18:38.246340 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:38.246267 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rg8qz_6ddcc417-9097-475a-924a-456207977712/manager/0.log" Jul 22 01:18:38.434327 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:38.434294 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8w8ss_776cedef-60e1-4cae-a066-545d90fd56a8/manager/0.log" Jul 22 01:18:38.674347 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:38.674317 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-j7rgz_f94935d9-4b4b-41df-934f-4208f1ad6386/manager/0.log" Jul 22 01:18:38.708058 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:38.708029 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-j7rgz_f94935d9-4b4b-41df-934f-4208f1ad6386/copy-manifests/0.log" Jul 22 01:18:39.805606 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:39.805572 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hgmfl_174e6bc7-959e-4306-ba1e-7d5d06bdb080/istio-proxy/0.log" Jul 22 01:18:39.982677 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:39.982631 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pqbfw_60abed7c-dc00-409b-9613-99580bb71748/istio-proxy/0.log" Jul 22 01:18:40.008119 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:40.008085 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pqbfw_60abed7c-dc00-409b-9613-99580bb71748/istio-proxy/1.log" Jul 22 01:18:40.318130 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:40.318095 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bd6f87dc6-xzhvg_e161a16f-e1b7-4a67-96ff-c073121d85d2/manager/0.log" Jul 22 01:18:41.295715 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:41.295686 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hgmfl_174e6bc7-959e-4306-ba1e-7d5d06bdb080/istio-proxy/0.log" Jul 22 01:18:41.480078 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:41.480048 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pqbfw_60abed7c-dc00-409b-9613-99580bb71748/istio-proxy/0.log" Jul 22 01:18:41.506699 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:41.506666 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pqbfw_60abed7c-dc00-409b-9613-99580bb71748/istio-proxy/1.log" Jul 22 01:18:45.455216 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:45.455186 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sllrw_d43ba888-ca31-462d-ba70-5ab908c4990f/migrator/0.log" Jul 22 01:18:45.476472 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:45.476445 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-sllrw_d43ba888-ca31-462d-ba70-5ab908c4990f/graceful-termination/0.log" Jul 22 01:18:46.923394 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:46.923367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mzrpx_b13067b2-8bb6-4faf-8818-e50de72f1db6/kube-multus-additional-cni-plugins/0.log" Jul 22 01:18:46.945138 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:46.945109 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mzrpx_b13067b2-8bb6-4faf-8818-e50de72f1db6/egress-router-binary-copy/0.log" Jul 22 01:18:46.965630 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:46.965602 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mzrpx_b13067b2-8bb6-4faf-8818-e50de72f1db6/cni-plugins/0.log" Jul 22 01:18:46.985827 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:46.985800 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mzrpx_b13067b2-8bb6-4faf-8818-e50de72f1db6/bond-cni-plugin/0.log" Jul 22 01:18:47.007240 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:47.007211 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mzrpx_b13067b2-8bb6-4faf-8818-e50de72f1db6/routeoverride-cni/0.log" Jul 22 01:18:47.031669 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:47.031630 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mzrpx_b13067b2-8bb6-4faf-8818-e50de72f1db6/whereabouts-cni-bincopy/0.log" Jul 22 01:18:47.059449 ip-10-0-131-63 kubenswrapper[2562]: I0722 01:18:47.059423 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mzrpx_b13067b2-8bb6-4faf-8818-e50de72f1db6/whereabouts-cni/0.log"