Jul 06 12:45:14.325328 ip-10-0-138-145 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 12:45:14.325344 ip-10-0-138-145 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 12:45:14.325372 ip-10-0-138-145 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 12:45:14.325708 ip-10-0-138-145 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 12:45:24.320106 ip-10-0-138-145 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 06 12:45:24.320124 ip-10-0-138-145 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5429e148fe0c4b6ea436934be287a0fa -- Jul 06 12:46:55.932914 ip-10-0-138-145 systemd[1]: Starting Kubernetes Kubelet... Jul 06 12:46:56.377559 ip-10-0-138-145 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 12:46:56.377559 ip-10-0-138-145 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 12:46:56.377559 ip-10-0-138-145 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 12:46:56.377559 ip-10-0-138-145 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 12:46:56.377559 ip-10-0-138-145 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 12:46:56.379624 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.379539 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 12:46:56.386254 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386238 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:46:56.386254 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386254 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386259 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386262 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386266 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386269 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386272 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386275 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386277 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386280 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386282 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386285 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386287 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386290 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386294 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386297 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386300 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386302 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386305 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386311 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386314 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:46:56.386319 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386316 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386319 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386322 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386325 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386327 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386330 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386333 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386335 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386338 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386340 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386343 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386345 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386348 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386350 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386352 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386355 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386357 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386360 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386362 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386365 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:46:56.386911 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386367 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386370 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386372 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386375 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386377 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386380 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386382 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386386 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386389 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386392 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386395 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386398 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386401 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386403 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386408 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386410 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386413 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386416 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386418 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:46:56.387387 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386421 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386423 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386425 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386428 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386430 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386433 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386436 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386438 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386441 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386444 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386446 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386449 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386453 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386457 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386459 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386461 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386464 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386466 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386469 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386471 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:46:56.387857 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386473 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386477 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386479 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386481 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386485 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386489 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386492 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386495 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386498 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386500 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386503 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386506 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386508 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386511 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386513 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386516 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386518 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386521 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.386523 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:46:56.388334 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387236 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387242 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387245 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387249 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387252 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387258 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387261 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387264 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387267 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387269 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387272 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387274 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387276 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387279 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387282 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387284 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387287 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387289 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387294 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387298 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:46:56.388848 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387301 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387304 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387307 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387309 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387312 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387314 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387317 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387319 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387322 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387324 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387326 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387329 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387331 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387334 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387336 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387339 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387341 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387344 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387348 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387351 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:46:56.389345 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387354 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387356 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387358 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387361 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387363 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387366 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387368 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387371 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387373 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387376 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387379 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387381 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387384 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387386 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387389 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387393 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387396 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387399 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387401 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387404 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:46:56.389852 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387406 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387409 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387411 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387414 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387417 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387419 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387422 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387424 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387429 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387433 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387436 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387440 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387443 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387445 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387448 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387450 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387453 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387455 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387457 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:46:56.390340 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387460 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387462 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387466 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387468 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387471 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387473 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387476 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387478 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387480 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387483 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387486 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387488 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387490 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387492 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387495 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387498 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387500 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387503 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387505 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387508 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:46:56.390814 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387910 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387916 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387920 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387924 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387928 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387930 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387933 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387935 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387938 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387940 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387943 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387945 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387949 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387952 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387956 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387958 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387962 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387964 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387967 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:46:56.391305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387970 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387973 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387975 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387978 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387980 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387983 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387986 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387989 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387992 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387994 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387996 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.387999 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388001 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388004 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388006 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388009 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388011 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388013 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388016 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388018 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388021 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:46:56.391770 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388023 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388026 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388028 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388031 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388033 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388035 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388039 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388041 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388044 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388046 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388048 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388051 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388053 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388056 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388058 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388061 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388063 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388065 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388068 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388071 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:46:56.392273 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388073 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388075 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388078 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388080 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388083 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388086 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388088 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388090 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388093 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388095 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388098 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388100 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388103 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388105 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388108 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388110 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388113 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388115 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388118 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388120 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:46:56.392767 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388123 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388125 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388128 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388130 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388133 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388135 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388137 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388140 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388142 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388145 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388148 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388150 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388153 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388159 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388161 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388164 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388167 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388170 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388174 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:46:56.393245 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388241 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388248 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388255 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388258 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388263 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388267 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388270 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388274 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388277 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388280 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388283 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388287 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388290 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388293 2568 flags.go:64] FLAG: --cgroup-root="" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388296 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388299 2568 flags.go:64] FLAG: --client-ca-file="" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388302 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388304 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388309 2568 flags.go:64] FLAG: --cluster-domain="" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388312 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388314 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388318 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388321 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388325 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 12:46:56.393756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388327 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388331 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388334 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388337 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388340 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388343 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388346 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388349 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388353 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388355 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388358 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388361 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388364 2568 flags.go:64] FLAG: --enable-server="true" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388367 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388371 2568 flags.go:64] FLAG: --event-burst="100" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388375 2568 flags.go:64] FLAG: --event-qps="50" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388377 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388380 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388383 2568 flags.go:64] FLAG: --eviction-hard="" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388387 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388390 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388395 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388398 2568 flags.go:64] FLAG: --eviction-soft="" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388400 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388403 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 12:46:56.394331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388406 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388409 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388411 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388414 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388417 2568 flags.go:64] FLAG: --feature-gates="" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388423 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388425 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388428 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388431 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388434 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388436 2568 flags.go:64] FLAG: --help="false" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388439 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388442 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388444 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388447 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388450 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388453 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388456 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388458 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388461 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388463 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388466 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388469 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388472 2568 flags.go:64] FLAG: --kube-reserved="" Jul 06 12:46:56.394959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388475 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388478 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388480 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388483 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388487 2568 flags.go:64] FLAG: --lock-file="" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388490 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388493 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388496 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388501 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388503 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388506 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388509 2568 flags.go:64] FLAG: --logging-format="text" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388512 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388515 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388517 2568 flags.go:64] FLAG: --manifest-url="" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388520 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388525 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388527 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388531 2568 flags.go:64] FLAG: --max-pods="110" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388534 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388537 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388541 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388544 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388547 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388549 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 12:46:56.395531 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388552 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388558 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388561 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388564 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388567 2568 flags.go:64] FLAG: --pod-cidr="" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388570 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388575 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388578 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388581 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388584 2568 flags.go:64] FLAG: --port="10250" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388587 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388591 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07c2a0e31728d817b" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388594 2568 flags.go:64] FLAG: --qos-reserved="" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388597 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388599 2568 flags.go:64] FLAG: --register-node="true" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388602 2568 flags.go:64] FLAG: --register-with-taints="" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388606 2568 flags.go:64] FLAG: --registry-burst="10" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388608 2568 flags.go:64] FLAG: --registry-qps="5" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388611 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388614 2568 flags.go:64] FLAG: --reserved-memory="" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388617 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388620 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388623 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388625 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388642 2568 flags.go:64] FLAG: --runonce="false" Jul 06 12:46:56.396162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388645 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388648 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388651 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388654 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388658 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388661 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388664 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388667 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388670 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388672 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388675 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388678 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388681 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388684 2568 flags.go:64] FLAG: --system-cgroups="" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388687 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388692 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388695 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388698 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388704 2568 flags.go:64] FLAG: --tls-min-version="" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388707 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388709 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388712 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388715 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388717 2568 flags.go:64] FLAG: --v="2" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388721 2568 flags.go:64] FLAG: --version="false" Jul 06 12:46:56.396793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388725 2568 flags.go:64] FLAG: --vmodule="" Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388728 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.388731 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388858 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388863 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388866 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388869 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388872 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388874 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388878 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388882 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388886 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388889 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388892 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388895 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388897 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388900 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388903 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388905 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:46:56.397432 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388908 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388911 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388914 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388917 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388920 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388923 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388926 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388928 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388931 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388933 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388935 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388938 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388940 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388942 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388945 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388948 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388950 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388952 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388955 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388957 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:46:56.397922 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388960 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388962 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388965 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388967 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388971 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388974 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388976 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388979 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388981 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388983 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388986 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388989 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388991 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388993 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388996 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.388999 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389001 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389003 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389006 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389009 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:46:56.398434 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389011 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389014 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389016 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389019 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389021 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389023 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389026 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389028 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389032 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389035 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389037 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389040 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389042 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389045 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389047 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389049 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389053 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389056 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389058 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:46:56.399259 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389061 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389063 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389066 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389068 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389071 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389073 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389075 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389078 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389082 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389086 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389089 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389092 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389095 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389098 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389101 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389103 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389106 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389108 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389110 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389113 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:46:56.399935 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389115 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389118 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389120 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.389122 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.389908 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.397333 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.397347 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.398203 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.398210 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398806 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398857 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398863 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398875 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398881 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:46:56.400442 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398886 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398890 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398894 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398899 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398903 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398907 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398911 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398948 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398954 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398961 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398970 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398976 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398982 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398988 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398992 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.398997 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399003 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399007 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399011 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399018 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:46:56.400934 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399022 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399026 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399037 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399041 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399048 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399052 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399056 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399092 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399096 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399100 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399104 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399108 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399112 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399116 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399121 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399130 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399134 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399138 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399142 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399146 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:46:56.401424 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399182 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399187 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399191 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399195 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399199 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399204 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399208 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399212 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399220 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399224 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399228 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399233 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399237 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399241 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399244 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399249 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399254 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399258 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399262 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399266 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:46:56.401932 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399271 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399279 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399284 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399288 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399292 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399296 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399300 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399304 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399308 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399312 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399317 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399321 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399325 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399358 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399367 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399372 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399376 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399380 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399385 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:46:56.402418 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399389 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399393 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399397 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399401 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399408 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399414 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399419 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399428 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399432 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399437 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399441 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399445 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399449 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399453 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.399458 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:46:56.402906 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.399466 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400114 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400129 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400132 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400135 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400138 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400141 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400144 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400147 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400149 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400152 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400154 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400158 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400160 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400163 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400165 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400168 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400170 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400173 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 12:46:56.403317 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400175 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400178 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400180 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400183 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400187 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400192 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400194 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400197 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400199 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400202 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400206 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400210 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400213 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400215 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400219 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400222 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400224 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400227 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400230 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400232 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 12:46:56.403802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400235 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400238 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400240 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400242 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400245 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400248 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400250 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400253 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400255 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400258 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400261 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400263 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400266 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400269 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400271 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400273 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400276 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400278 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400281 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400283 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 12:46:56.404305 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400285 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400288 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400290 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400293 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400296 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400298 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400301 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400304 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400306 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400309 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400311 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400314 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400316 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400319 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400322 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400324 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400326 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400329 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400331 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400334 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 12:46:56.404906 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400338 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400341 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400344 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400347 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400350 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400352 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400355 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400357 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400360 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400363 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400366 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400368 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400371 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400374 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400376 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400379 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400382 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400385 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400387 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400391 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 12:46:56.405403 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:56.400393 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 12:46:56.405940 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.400399 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 12:46:56.405940 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.400518 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 12:46:56.405940 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.402905 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 12:46:56.405940 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.404576 2568 server.go:1019] "Starting client certificate rotation" Jul 06 12:46:56.405940 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.404694 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 12:46:56.405940 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.404737 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 12:46:56.407353 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.407339 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 12:46:56.409388 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.409362 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 12:46:56.424510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.424492 2568 log.go:25] "Validated CRI v1 runtime API" Jul 06 12:46:56.428599 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.428582 2568 log.go:25] "Validated CRI v1 image API" Jul 06 12:46:56.429418 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.429405 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 12:46:56.434026 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.434005 2568 fs.go:135] Filesystem UUIDs: map[1cb51029-d2f6-4c99-ac10-2757bdae427d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7c4a3a14-c945-4b02-87db-1da4d5702c3f:/dev/nvme0n1p4] Jul 06 12:46:56.434077 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.434027 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 12:46:56.440768 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.440750 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 12:46:56.441255 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.441141 2568 manager.go:216] Machine: {Timestamp:2026-07-06 12:46:56.438728177 +0000 UTC m=+0.379569027 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098797 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23532783c91a3e5cb6421fbd019028 SystemUUID:ec235327-83c9-1a3e-5cb6-421fbd019028 BootID:5429e148-fe0c-4b6e-a436-934be287a0fa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:0c:55:ce:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:0c:55:ce:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:68:e4:0c:2c:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 12:46:56.441255 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.441253 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 12:46:56.441361 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.441331 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 12:46:56.444995 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.444968 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 12:46:56.445133 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.444998 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-145.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 12:46:56.445913 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.445903 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 12:46:56.445950 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.445915 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 12:46:56.445950 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.445922 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 12:46:56.447545 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.447535 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 12:46:56.449806 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.449797 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 12:46:56.454492 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.454481 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 12:46:56.454611 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.454602 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 12:46:56.454654 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.454643 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 06 12:46:56.454688 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.454656 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 12:46:56.454688 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.454666 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 12:46:56.454688 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.454681 2568 kubelet.go:398] "Adding apiserver pod source" Jul 06 12:46:56.454793 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.454690 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 12:46:56.456648 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.456591 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 12:46:56.456751 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.456651 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 12:46:56.459685 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.459668 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 12:46:56.462233 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.462220 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 12:46:56.462297 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.462250 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 12:46:56.463716 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463703 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 12:46:56.463753 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463727 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 12:46:56.463753 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463734 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 12:46:56.463753 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463740 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 12:46:56.463753 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463750 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 12:46:56.463860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463755 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 12:46:56.463860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463765 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 12:46:56.463860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463770 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 12:46:56.463860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463777 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 12:46:56.463860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463783 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 12:46:56.463860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463793 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 12:46:56.463860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.463799 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 12:46:56.464938 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.464928 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 12:46:56.464969 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.464939 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 12:46:56.468335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.468309 2568 server.go:1268] "Started kubelet" Jul 06 12:46:56.468439 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.468388 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 12:46:56.468494 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.468450 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 12:46:56.468553 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.468501 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 12:46:56.470424 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.470404 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 12:46:56.470604 ip-10-0-138-145 systemd[1]: Started Kubernetes Kubelet. Jul 06 12:46:56.470735 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.470696 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 06 12:46:56.472839 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.472810 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-145.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 12:46:56.473677 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.473648 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 12:46:56.473750 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.473649 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 12:46:56.474433 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.474416 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 12:46:56.478014 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.477999 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 12:46:56.478086 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.478012 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 12:46:56.478261 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.478243 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:56.478823 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.478794 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 12:46:56.478823 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.478818 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 12:46:56.479034 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.478793 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 12:46:56.479034 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.478932 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 06 12:46:56.479034 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.478940 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 06 12:46:56.479171 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479127 2568 factory.go:55] Registering systemd factory Jul 06 12:46:56.479171 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479142 2568 factory.go:223] Registration of the systemd container factory successfully Jul 06 12:46:56.479291 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.477911 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-145.ec2.internal.18bfb41d8c9ea47c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-145.ec2.internal,UID:ip-10-0-138-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-145.ec2.internal,},FirstTimestamp:2026-07-06 12:46:56.46826406 +0000 UTC m=+0.409104913,LastTimestamp:2026-07-06 12:46:56.46826406 +0000 UTC m=+0.409104913,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-145.ec2.internal,}" Jul 06 12:46:56.479371 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479359 2568 factory.go:153] Registering CRI-O factory Jul 06 12:46:56.479412 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479372 2568 factory.go:223] Registration of the crio container factory successfully Jul 06 12:46:56.479446 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479414 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 12:46:56.479446 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479432 2568 factory.go:103] Registering Raw factory Jul 06 12:46:56.479446 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479442 2568 manager.go:1196] Started watching for new ooms in manager Jul 06 12:46:56.479994 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.479981 2568 manager.go:349] Starting recovery of all containers Jul 06 12:46:56.484343 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.484319 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 12:46:56.484414 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.484340 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kjgql" Jul 06 12:46:56.484487 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.484465 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 12:46:56.493598 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.493486 2568 manager.go:354] Recovery completed Jul 06 12:46:56.493833 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.493812 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kjgql" Jul 06 12:46:56.497866 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.497853 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:46:56.499432 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.499414 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:46:56.499508 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.499447 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:46:56.499508 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.499470 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:46:56.500089 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.500075 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 12:46:56.500089 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.500088 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 12:46:56.500208 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.500094 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:46:56.500208 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.500103 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 12:46:56.500772 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.500754 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:46:56.500827 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.500778 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:46:56.500827 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.500787 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:46:56.501886 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.501818 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-145.ec2.internal.18bfb41d8e7a31e2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-145.ec2.internal,UID:ip-10-0-138-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-145.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-145.ec2.internal,},FirstTimestamp:2026-07-06 12:46:56.499429858 +0000 UTC m=+0.440270718,LastTimestamp:2026-07-06 12:46:56.499429858 +0000 UTC m=+0.440270718,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-145.ec2.internal,}" Jul 06 12:46:56.502071 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.502059 2568 policy_none.go:49] "None policy: Start" Jul 06 12:46:56.502124 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.502074 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 12:46:56.502124 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.502084 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 12:46:56.504063 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.504052 2568 policy_none.go:47] "Start" Jul 06 12:46:56.539221 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.539205 2568 manager.go:337] "Starting Device Plugin manager" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.539242 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.539255 2568 server.go:87] "Starting device plugin registration server" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.539452 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.539464 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.539542 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.539611 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.539622 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.540136 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 12:46:56.563181 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.540182 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:56.583003 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.582975 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 12:46:56.584281 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.584234 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 12:46:56.584281 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.584264 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 12:46:56.584418 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.584284 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 12:46:56.584418 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.584313 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 12:46:56.586832 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.586817 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:46:56.639757 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.639737 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:46:56.640527 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.640511 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:46:56.640613 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.640547 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:46:56.640613 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.640561 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:46:56.640613 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.640590 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.647264 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.647250 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.647345 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.647272 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-145.ec2.internal\": node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:56.666155 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.666132 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:56.685070 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.685025 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal"] Jul 06 12:46:56.685128 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.685109 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:46:56.685889 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.685875 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:46:56.685971 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.685905 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:46:56.685971 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.685920 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:46:56.688292 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.688278 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:46:56.688407 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.688390 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.688451 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.688423 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:46:56.688980 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.688964 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:46:56.689108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.688996 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:46:56.689108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.689009 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:46:56.689108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.688964 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:46:56.689108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.689078 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:46:56.689108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.689090 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:46:56.691319 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.691302 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.691396 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.691335 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 12:46:56.691972 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.691958 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientMemory" Jul 06 12:46:56.692030 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.691983 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 12:46:56.692030 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.691996 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeHasSufficientPID" Jul 06 12:46:56.726228 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.726201 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-145.ec2.internal\" not found" node="ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.730510 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.730494 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-145.ec2.internal\" not found" node="ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.766684 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.766665 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:56.780761 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.780743 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2b28ef55d482c311313ef51a865f9fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal\" (UID: \"b2b28ef55d482c311313ef51a865f9fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.780811 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.780769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2b28ef55d482c311313ef51a865f9fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal\" (UID: \"b2b28ef55d482c311313ef51a865f9fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.780811 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.780787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36c64bbcbb7e33685ecee28ec1d660b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-145.ec2.internal\" (UID: \"36c64bbcbb7e33685ecee28ec1d660b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.867158 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.867096 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:56.881450 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.881417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2b28ef55d482c311313ef51a865f9fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal\" (UID: \"b2b28ef55d482c311313ef51a865f9fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.881510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.881456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2b28ef55d482c311313ef51a865f9fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal\" (UID: \"b2b28ef55d482c311313ef51a865f9fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.881510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.881482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36c64bbcbb7e33685ecee28ec1d660b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-145.ec2.internal\" (UID: \"36c64bbcbb7e33685ecee28ec1d660b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.881578 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.881507 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2b28ef55d482c311313ef51a865f9fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal\" (UID: \"b2b28ef55d482c311313ef51a865f9fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.881578 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.881520 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36c64bbcbb7e33685ecee28ec1d660b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-145.ec2.internal\" (UID: \"36c64bbcbb7e33685ecee28ec1d660b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.881578 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:56.881524 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2b28ef55d482c311313ef51a865f9fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal\" (UID: \"b2b28ef55d482c311313ef51a865f9fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:56.967858 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:56.967834 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:57.029319 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.029290 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:57.034489 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.034364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" Jul 06 12:46:57.068861 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:57.068835 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:57.169369 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:57.169298 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:57.269902 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:57.269870 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:57.370426 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:57.370386 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:57.403916 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.403882 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 12:46:57.404461 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.404019 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 12:46:57.471704 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:57.471465 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-145.ec2.internal\" not found" Jul 06 12:46:57.500463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.500432 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 12:41:56 +0000 UTC" deadline="2027-12-03 10:14:53.332831925 +0000 UTC" Jul 06 12:46:57.500463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.500459 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12357h27m55.832375931s" Jul 06 12:46:57.540349 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.540328 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:46:57.577837 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.577817 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:46:57.578619 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.578603 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" Jul 06 12:46:57.586044 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:57.586014 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c64bbcbb7e33685ecee28ec1d660b6.slice/crio-0595058970b305c3942040a94a41a8802dd5d3b1f5ba3b66cf9ffc9c80515a56 WatchSource:0}: Error finding container 0595058970b305c3942040a94a41a8802dd5d3b1f5ba3b66cf9ffc9c80515a56: Status 404 returned error can't find the container with id 0595058970b305c3942040a94a41a8802dd5d3b1f5ba3b66cf9ffc9c80515a56 Jul 06 12:46:57.591741 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.591728 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 12:46:57.599784 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.599770 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 12:46:57.600614 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.600603 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" Jul 06 12:46:57.611838 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.611823 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 12:46:57.612668 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:57.612651 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b28ef55d482c311313ef51a865f9fa.slice/crio-6dad3e5d78e33b1d6f9953e45bc9080a93785c2e11d55a46f807cbd6e0219e8d WatchSource:0}: Error finding container 6dad3e5d78e33b1d6f9953e45bc9080a93785c2e11d55a46f807cbd6e0219e8d: Status 404 returned error can't find the container with id 6dad3e5d78e33b1d6f9953e45bc9080a93785c2e11d55a46f807cbd6e0219e8d Jul 06 12:46:57.884129 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:57.884062 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:46:58.455952 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.455923 2568 apiserver.go:52] "Watching apiserver" Jul 06 12:46:58.463112 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.463091 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 12:46:58.463460 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.463440 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9jnkb","openshift-multus/multus-mxk6l","openshift-network-diagnostics/network-check-target-w6dgr","openshift-network-operator/iptables-alerter-g86l7","kube-system/konnectivity-agent-r7g6r","kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal","openshift-dns/node-resolver-mbb7x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal","openshift-multus/network-metrics-daemon-8n477","openshift-ovn-kubernetes/ovnkube-node-8cmpd","kube-system/global-pull-secret-syncer-gc9tb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2","openshift-cluster-node-tuning-operator/tuned-rg5g8","openshift-image-registry/node-ca-9x5md"] Jul 06 12:46:58.465823 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.465804 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.468007 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.467990 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.468432 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.468417 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qztz5\"" Jul 06 12:46:58.468797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.468774 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 12:46:58.469063 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.469045 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 12:46:58.470694 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.470679 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:46:58.470926 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.470899 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:46:58.471007 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.470958 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8cszw\"" Jul 06 12:46:58.471007 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.470957 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:46:58.471007 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.470962 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 12:46:58.472748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.472733 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.476121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.474690 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.476121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.474691 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:46:58.476121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.475286 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 12:46:58.476121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.475484 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 12:46:58.476298 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.476147 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d9h5k\"" Jul 06 12:46:58.476954 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.476930 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t8rcr\"" Jul 06 12:46:58.477068 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.477050 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 12:46:58.477744 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.477206 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.477744 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.477610 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 12:46:58.479523 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.479254 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 12:46:58.479523 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.479368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:46:58.479523 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.479422 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:46:58.480802 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.480754 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 12:46:58.481404 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.481384 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 12:46:58.481762 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.481747 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 12:46:58.482356 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.482179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.483567 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.483168 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l86zk\"" Jul 06 12:46:58.484735 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.484717 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.486295 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.486277 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 12:46:58.486422 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.486411 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 12:46:58.486573 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.486562 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 12:46:58.486724 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.486704 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sphd2\"" Jul 06 12:46:58.486813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.486798 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 12:46:58.486905 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.486891 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 12:46:58.487263 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487248 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.487333 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487323 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 12:46:58.487801 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db4973a1-3cc2-4a94-aa2b-18d04d82a7d9-agent-certs\") pod \"konnectivity-agent-r7g6r\" (UID: \"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9\") " pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.487887 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487814 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-systemd\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.487887 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:46:58.487887 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdn4v\" (UniqueName: \"kubernetes.io/projected/efdb6ed1-7693-4154-8cd4-94005582afff-kube-api-access-qdn4v\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.487887 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487880 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-var-lib-kubelet\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tr6b\" (UniqueName: \"kubernetes.io/projected/d42744ee-0140-41b4-afba-64fd0483ee84-kube-api-access-2tr6b\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:46:58.488073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487943 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-system-cni-dir\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysctl-d\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487980 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zngrq\"" Jul 06 12:46:58.488073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.487999 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-tuned\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488034 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnrqz\" (UniqueName: \"kubernetes.io/projected/2d398316-6b06-4e5e-8e3d-174be9a72a8b-kube-api-access-pnrqz\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db4973a1-3cc2-4a94-aa2b-18d04d82a7d9-konnectivity-ca\") pod \"konnectivity-agent-r7g6r\" (UID: \"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9\") " pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-run\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-sys\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488184 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-modprobe-d\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-host\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76e28400-b5b5-423d-9a3c-f4731f5e8121-tmp\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488280 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cnibin\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cc5d81-f939-4834-b1f8-80f4d2e090ce-host-slash\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488329 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysctl-conf\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhglf\" (UniqueName: \"kubernetes.io/projected/76e28400-b5b5-423d-9a3c-f4731f5e8121-kube-api-access-zhglf\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488366 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efdb6ed1-7693-4154-8cd4-94005582afff-serviceca\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488390 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2168b17-6206-410a-a930-f665ddda267a-hosts-file\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488430 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-os-release\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488460 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488481 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2168b17-6206-410a-a930-f665ddda267a-tmp-dir\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488518 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2qn\" (UniqueName: \"kubernetes.io/projected/d2168b17-6206-410a-a930-f665ddda267a-kube-api-access-cb2qn\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18cc5d81-f939-4834-b1f8-80f4d2e090ce-iptables-alerter-script\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488559 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-lib-modules\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysconfig\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-kubernetes\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efdb6ed1-7693-4154-8cd4-94005582afff-host\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.488863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.488667 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzdn\" (UniqueName: \"kubernetes.io/projected/18cc5d81-f939-4834-b1f8-80f4d2e090ce-kube-api-access-xtzdn\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.489528 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.489514 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.489620 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.489597 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:46:58.490878 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.490851 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 12:46:58.490963 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.490905 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 12:46:58.491024 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.490999 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 12:46:58.491154 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.491130 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 12:46:58.491259 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.491245 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xnllj\"" Jul 06 12:46:58.491362 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.491349 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 12:46:58.491604 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.491514 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 12:46:58.491869 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.491852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.493828 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.493811 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 12:46:58.493900 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.493834 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 12:46:58.494679 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.494045 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 12:46:58.494679 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.494081 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4z46g\"" Jul 06 12:46:58.494679 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.494176 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 12:46:58.545891 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.545868 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9vb6c" Jul 06 12:46:58.552723 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.552699 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9vb6c" Jul 06 12:46:58.579925 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.579904 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 12:46:58.590279 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-cnibin\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.590387 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590289 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-kubelet\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.590387 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysctl-d\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.590387 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-tuned\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.590527 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-ovn\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.590527 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-registration-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.590527 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590463 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-cni-multus\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.590527 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46062402-20c6-4730-8520-b1a94ef3c11a-kubelet-config\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.590527 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590510 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysctl-d\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db4973a1-3cc2-4a94-aa2b-18d04d82a7d9-konnectivity-ca\") pod \"konnectivity-agent-r7g6r\" (UID: \"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9\") " pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-run\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-sys\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590689 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-sys\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590689 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590724 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-run\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590733 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.590773 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590766 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-cni-netd\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590825 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndfg\" (UniqueName: \"kubernetes.io/projected/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-kube-api-access-tndfg\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76e28400-b5b5-423d-9a3c-f4731f5e8121-tmp\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cnibin\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhglf\" (UniqueName: \"kubernetes.io/projected/76e28400-b5b5-423d-9a3c-f4731f5e8121-kube-api-access-zhglf\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efdb6ed1-7693-4154-8cd4-94005582afff-serviceca\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.590999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2168b17-6206-410a-a930-f665ddda267a-hosts-file\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591051 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-slash\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591077 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591098 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db4973a1-3cc2-4a94-aa2b-18d04d82a7d9-konnectivity-ca\") pod \"konnectivity-agent-r7g6r\" (UID: \"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9\") " pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgpw\" (UniqueName: \"kubernetes.io/projected/f46d4222-907b-4cb6-8217-38e8228ee9d6-kube-api-access-8vgpw\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.591141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-cni-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.591664 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-multus-certs\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.591664 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.591664 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2168b17-6206-410a-a930-f665ddda267a-tmp-dir\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.591664 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2qn\" (UniqueName: \"kubernetes.io/projected/d2168b17-6206-410a-a930-f665ddda267a-kube-api-access-cb2qn\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.591664 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591270 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.591857 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18cc5d81-f939-4834-b1f8-80f4d2e090ce-iptables-alerter-script\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.591902 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-systemd\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.591902 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.591875 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:46:58.591989 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.591953 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:46:59.09193312 +0000 UTC m=+3.032773973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:46:58.592048 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592033 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cnibin\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.592128 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.591880 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovnkube-script-lib\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.592175 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592152 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-os-release\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.592220 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592180 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-cni-binary-copy\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.592220 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-kubernetes\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.592307 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzdn\" (UniqueName: \"kubernetes.io/projected/18cc5d81-f939-4834-b1f8-80f4d2e090ce-kube-api-access-xtzdn\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.592307 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-run-netns\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.592307 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-socket-dir-parent\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.592307 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592291 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-netns\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.592416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db4973a1-3cc2-4a94-aa2b-18d04d82a7d9-agent-certs\") pod \"konnectivity-agent-r7g6r\" (UID: \"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9\") " pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.592416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdn4v\" (UniqueName: \"kubernetes.io/projected/efdb6ed1-7693-4154-8cd4-94005582afff-kube-api-access-qdn4v\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.592416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-kubelet\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.592416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-node-log\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.592416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592376 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18cc5d81-f939-4834-b1f8-80f4d2e090ce-iptables-alerter-script\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.592416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592390 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-conf-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.592584 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-var-lib-kubelet\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.592584 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592472 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-var-lib-kubelet\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.592584 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tr6b\" (UniqueName: \"kubernetes.io/projected/d42744ee-0140-41b4-afba-64fd0483ee84-kube-api-access-2tr6b\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:46:58.592584 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-system-cni-dir\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.592584 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592572 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/efdb6ed1-7693-4154-8cd4-94005582afff-serviceca\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.592747 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.592747 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-env-overrides\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.592747 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592654 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d2168b17-6206-410a-a930-f665ddda267a-hosts-file\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.592747 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-device-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.592747 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-sys-fs\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.592871 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592750 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-system-cni-dir\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.592871 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592757 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-etc-kubernetes\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.592871 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.592871 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnrqz\" (UniqueName: \"kubernetes.io/projected/2d398316-6b06-4e5e-8e3d-174be9a72a8b-kube-api-access-pnrqz\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.592871 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592848 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lc7j\" (UniqueName: \"kubernetes.io/projected/1af5ef28-47db-42d8-8bd0-24b289863efc-kube-api-access-8lc7j\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.593013 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592872 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-system-cni-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.593013 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592893 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-hostroot\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.593013 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592913 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46062402-20c6-4730-8520-b1a94ef3c11a-dbus\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.593013 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.592971 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-kubernetes\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.593151 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593133 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.593278 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593253 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.593554 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.593554 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.593554 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovnkube-config\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-etc-selinux\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593655 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2168b17-6206-410a-a930-f665ddda267a-tmp-dir\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593662 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-k8s-cni-cncf-io\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-modprobe-d\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593682 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" event={"ID":"b2b28ef55d482c311313ef51a865f9fa","Type":"ContainerStarted","Data":"6dad3e5d78e33b1d6f9953e45bc9080a93785c2e11d55a46f807cbd6e0219e8d"} Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-host\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cc5d81-f939-4834-b1f8-80f4d2e090ce-host-slash\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593774 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cc5d81-f939-4834-b1f8-80f4d2e090ce-host-slash\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593778 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-host\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.593813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-daemon-config\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-modprobe-d\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-var-lib-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-log-socket\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.593980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysctl-conf\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-systemd-units\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594013 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d398316-6b06-4e5e-8e3d-174be9a72a8b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594047 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-cni-bin\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-os-release\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594104 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-etc-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594116 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysctl-conf\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594152 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d398316-6b06-4e5e-8e3d-174be9a72a8b-os-release\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-cni-bin\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594220 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-lib-modules\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594253 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysconfig\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efdb6ed1-7693-4154-8cd4-94005582afff-host\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-socket-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594341 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-lib-modules\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-systemd\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594380 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/efdb6ed1-7693-4154-8cd4-94005582afff-host\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594473 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-sysconfig\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.594988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.594521 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-systemd\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.595520 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.595494 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db4973a1-3cc2-4a94-aa2b-18d04d82a7d9-agent-certs\") pod \"konnectivity-agent-r7g6r\" (UID: \"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9\") " pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.595709 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.595694 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76e28400-b5b5-423d-9a3c-f4731f5e8121-etc-tuned\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.596988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.596967 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76e28400-b5b5-423d-9a3c-f4731f5e8121-tmp\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.597271 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.597248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" event={"ID":"36c64bbcbb7e33685ecee28ec1d660b6","Type":"ContainerStarted","Data":"0595058970b305c3942040a94a41a8802dd5d3b1f5ba3b66cf9ffc9c80515a56"} Jul 06 12:46:58.601121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.601089 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhglf\" (UniqueName: \"kubernetes.io/projected/76e28400-b5b5-423d-9a3c-f4731f5e8121-kube-api-access-zhglf\") pod \"tuned-rg5g8\" (UID: \"76e28400-b5b5-423d-9a3c-f4731f5e8121\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.601700 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.601663 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:46:58.601700 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.601685 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:46:58.601700 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.601698 2568 projected.go:196] Error preparing data for projected volume kube-api-access-svn9w for pod openshift-network-diagnostics/network-check-target-w6dgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:46:58.601859 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.601747 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w podName:f568bda6-dc07-4f9b-9921-32d0dc051147 nodeName:}" failed. No retries permitted until 2026-07-06 12:46:59.101730691 +0000 UTC m=+3.042571542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-svn9w" (UniqueName: "kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w") pod "network-check-target-w6dgr" (UID: "f568bda6-dc07-4f9b-9921-32d0dc051147") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:46:58.602351 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.602330 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2qn\" (UniqueName: \"kubernetes.io/projected/d2168b17-6206-410a-a930-f665ddda267a-kube-api-access-cb2qn\") pod \"node-resolver-mbb7x\" (UID: \"d2168b17-6206-410a-a930-f665ddda267a\") " pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.603960 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.603925 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnrqz\" (UniqueName: \"kubernetes.io/projected/2d398316-6b06-4e5e-8e3d-174be9a72a8b-kube-api-access-pnrqz\") pod \"multus-additional-cni-plugins-9jnkb\" (UID: \"2d398316-6b06-4e5e-8e3d-174be9a72a8b\") " pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.604553 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.604524 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tr6b\" (UniqueName: \"kubernetes.io/projected/d42744ee-0140-41b4-afba-64fd0483ee84-kube-api-access-2tr6b\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:46:58.604771 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.604757 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzdn\" (UniqueName: \"kubernetes.io/projected/18cc5d81-f939-4834-b1f8-80f4d2e090ce-kube-api-access-xtzdn\") pod \"iptables-alerter-g86l7\" (UID: \"18cc5d81-f939-4834-b1f8-80f4d2e090ce\") " pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.605829 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.605810 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdn4v\" (UniqueName: \"kubernetes.io/projected/efdb6ed1-7693-4154-8cd4-94005582afff-kube-api-access-qdn4v\") pod \"node-ca-9x5md\" (UID: \"efdb6ed1-7693-4154-8cd4-94005582afff\") " pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.695665 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695620 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndfg\" (UniqueName: \"kubernetes.io/projected/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-kube-api-access-tndfg\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.695797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-slash\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.695797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.695797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgpw\" (UniqueName: \"kubernetes.io/projected/f46d4222-907b-4cb6-8217-38e8228ee9d6-kube-api-access-8vgpw\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.695797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-cni-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.695797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-multus-certs\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.695797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695778 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-slash\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.695797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-systemd\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695842 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-systemd\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695852 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-cni-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695883 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-multus-certs\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovnkube-script-lib\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-os-release\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-cni-binary-copy\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695961 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-run-netns\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-socket-dir-parent\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-netns\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.695996 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-os-release\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-kubelet\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696028 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-run-netns\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696044 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-node-log\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696067 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-conf-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696080 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-socket-dir-parent\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696093 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696095 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-kubelet\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696164 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696067 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-netns\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696131 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-env-overrides\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696139 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-conf-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696143 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-node-log\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-device-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696177 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-sys-fs\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-etc-kubernetes\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696222 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-sys-fs\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lc7j\" (UniqueName: \"kubernetes.io/projected/1af5ef28-47db-42d8-8bd0-24b289863efc-kube-api-access-8lc7j\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696260 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-system-cni-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696273 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-device-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-hostroot\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46062402-20c6-4730-8520-b1a94ef3c11a-dbus\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovnkube-config\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-etc-selinux\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.696974 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-k8s-cni-cncf-io\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696445 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovnkube-script-lib\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696478 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-etc-kubernetes\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696481 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-cni-binary-copy\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696502 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-system-cni-dir\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-daemon-config\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-var-lib-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696538 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-hostroot\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696537 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-env-overrides\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696555 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46062402-20c6-4730-8520-b1a94ef3c11a-dbus\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696606 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-run-k8s-cni-cncf-io\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.696642 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-log-socket\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696682 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696692 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:58.696697 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret podName:46062402-20c6-4730-8520-b1a94ef3c11a nodeName:}" failed. No retries permitted until 2026-07-06 12:46:59.196682155 +0000 UTC m=+3.137523005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret") pod "global-pull-secret-syncer-gc9tb" (UID: "46062402-20c6-4730-8520-b1a94ef3c11a") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:46:58.697748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696713 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696746 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-systemd-units\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696752 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-log-socket\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696652 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-etc-selinux\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696808 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-cni-bin\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-cni-bin\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696832 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-var-lib-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-etc-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696868 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-systemd-units\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-cni-bin\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696891 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovnkube-config\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696903 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-etc-openvswitch\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-socket-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696945 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-cni-bin\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.696995 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-cnibin\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-kubelet\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697042 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-ovn\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697060 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-multus-daemon-config\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-registration-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697067 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-cnibin\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-cni-multus\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697097 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-kubelet\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697094 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-socket-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697110 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-run-ovn\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46062402-20c6-4730-8520-b1a94ef3c11a-kubelet-config\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697134 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af5ef28-47db-42d8-8bd0-24b289863efc-registration-dir\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697141 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-host-var-lib-cni-multus\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697150 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46062402-20c6-4730-8520-b1a94ef3c11a-kubelet-config\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-cni-netd\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697179 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697223 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.697231 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46d4222-907b-4cb6-8217-38e8228ee9d6-host-cni-netd\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.698965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.698169 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46d4222-907b-4cb6-8217-38e8228ee9d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.706193 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.706148 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndfg\" (UniqueName: \"kubernetes.io/projected/6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb-kube-api-access-tndfg\") pod \"multus-mxk6l\" (UID: \"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb\") " pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.710561 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.710537 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lc7j\" (UniqueName: \"kubernetes.io/projected/1af5ef28-47db-42d8-8bd0-24b289863efc-kube-api-access-8lc7j\") pod \"aws-ebs-csi-driver-node-kthn2\" (UID: \"1af5ef28-47db-42d8-8bd0-24b289863efc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.710752 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.710733 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgpw\" (UniqueName: \"kubernetes.io/projected/f46d4222-907b-4cb6-8217-38e8228ee9d6-kube-api-access-8vgpw\") pod \"ovnkube-node-8cmpd\" (UID: \"f46d4222-907b-4cb6-8217-38e8228ee9d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.777611 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.777575 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbb7x" Jul 06 12:46:58.787516 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.787494 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" Jul 06 12:46:58.796541 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.796520 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g86l7" Jul 06 12:46:58.801985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.801967 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:46:58.808302 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.808284 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9x5md" Jul 06 12:46:58.814658 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.814619 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" Jul 06 12:46:58.821046 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.821025 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mxk6l" Jul 06 12:46:58.828456 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.828442 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:46:58.834806 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.834789 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" Jul 06 12:46:58.929668 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:58.929645 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 12:46:59.099463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.099386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:46:59.099624 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.099581 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:46:59.099713 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.099673 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:00.09965124 +0000 UTC m=+4.040492079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:46:59.200199 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.200164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:59.200322 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.200217 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:46:59.200322 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.200312 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:46:59.200405 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.200355 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:46:59.200405 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.200368 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:46:59.200405 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.200377 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret podName:46062402-20c6-4730-8520-b1a94ef3c11a nodeName:}" failed. No retries permitted until 2026-07-06 12:47:00.200357981 +0000 UTC m=+4.141198822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret") pod "global-pull-secret-syncer-gc9tb" (UID: "46062402-20c6-4730-8520-b1a94ef3c11a") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:46:59.200405 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.200380 2568 projected.go:196] Error preparing data for projected volume kube-api-access-svn9w for pod openshift-network-diagnostics/network-check-target-w6dgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:46:59.200556 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.200420 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w podName:f568bda6-dc07-4f9b-9921-32d0dc051147 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:00.200407315 +0000 UTC m=+4.141248168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-svn9w" (UniqueName: "kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w") pod "network-check-target-w6dgr" (UID: "f568bda6-dc07-4f9b-9921-32d0dc051147") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:46:59.555766 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.555712 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 12:41:58 +0000 UTC" deadline="2028-02-15 17:28:14.29036695 +0000 UTC" Jul 06 12:46:59.556121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.555768 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14140h41m14.734602893s" Jul 06 12:46:59.564455 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:59.564423 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d398316_6b06_4e5e_8e3d_174be9a72a8b.slice/crio-adc17b0b96bdb73a03df0d74c0ee0ec36ade7dbf1882785d49d50c348d76f24f WatchSource:0}: Error finding container adc17b0b96bdb73a03df0d74c0ee0ec36ade7dbf1882785d49d50c348d76f24f: Status 404 returned error can't find the container with id adc17b0b96bdb73a03df0d74c0ee0ec36ade7dbf1882785d49d50c348d76f24f Jul 06 12:46:59.566470 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:59.566393 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdb6ed1_7693_4154_8cd4_94005582afff.slice/crio-dfea2e2ff1a6b30ca0cdd216c80c3473ae5fd5e28e2f5a67528329ddfd3c2b9e WatchSource:0}: Error finding container dfea2e2ff1a6b30ca0cdd216c80c3473ae5fd5e28e2f5a67528329ddfd3c2b9e: Status 404 returned error can't find the container with id dfea2e2ff1a6b30ca0cdd216c80c3473ae5fd5e28e2f5a67528329ddfd3c2b9e Jul 06 12:46:59.569142 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:46:59.568972 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af5ef28_47db_42d8_8bd0_24b289863efc.slice/crio-998ef73781ece39ae2695b4c37d9d340a6532631040366839280bcfb0fd1a87f WatchSource:0}: Error finding container 998ef73781ece39ae2695b4c37d9d340a6532631040366839280bcfb0fd1a87f: Status 404 returned error can't find the container with id 998ef73781ece39ae2695b4c37d9d340a6532631040366839280bcfb0fd1a87f Jul 06 12:46:59.584748 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.584724 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:46:59.584876 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:46:59.584854 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:46:59.599599 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.599466 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g86l7" event={"ID":"18cc5d81-f939-4834-b1f8-80f4d2e090ce","Type":"ContainerStarted","Data":"8b3c7fd8d7b00ec95da3fd7a7e2f50d4316f26859ca17bd7739a17f9cbe43aae"} Jul 06 12:46:59.602450 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.602426 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbb7x" event={"ID":"d2168b17-6206-410a-a930-f665ddda267a","Type":"ContainerStarted","Data":"62953aa72da9e4862edbb1bc707413fa66e633607900ae61ce1cb11082693ce2"} Jul 06 12:46:59.606968 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.606946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r7g6r" event={"ID":"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9","Type":"ContainerStarted","Data":"fef1bb940e5380b02b7fb63801edcfb9a2dea36980d6af1dce070640da902ec1"} Jul 06 12:46:59.612479 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.612410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mxk6l" event={"ID":"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb","Type":"ContainerStarted","Data":"187eabfa664f4d626f3d8841932909f6374e9abf239547735649b4137e27c006"} Jul 06 12:46:59.613387 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.613364 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" event={"ID":"1af5ef28-47db-42d8-8bd0-24b289863efc","Type":"ContainerStarted","Data":"998ef73781ece39ae2695b4c37d9d340a6532631040366839280bcfb0fd1a87f"} Jul 06 12:46:59.614334 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.614313 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9x5md" event={"ID":"efdb6ed1-7693-4154-8cd4-94005582afff","Type":"ContainerStarted","Data":"dfea2e2ff1a6b30ca0cdd216c80c3473ae5fd5e28e2f5a67528329ddfd3c2b9e"} Jul 06 12:46:59.615380 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.615361 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"bdf6951e78d8306aad11d405cb9462a73ff9dc36f1f6d62b95c8d2c3141466b3"} Jul 06 12:46:59.616367 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.616347 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" event={"ID":"76e28400-b5b5-423d-9a3c-f4731f5e8121","Type":"ContainerStarted","Data":"e1f7197ad92d324215d03ae4b00b877666718b78b8dfffe1d93a526878b5345b"} Jul 06 12:46:59.617247 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:46:59.617229 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerStarted","Data":"adc17b0b96bdb73a03df0d74c0ee0ec36ade7dbf1882785d49d50c348d76f24f"} Jul 06 12:47:00.108877 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.108803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:00.109052 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.108979 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:00.109114 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.109056 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:02.109035324 +0000 UTC m=+6.049876180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:00.209350 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.209313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:00.209519 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.209408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:00.209580 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.209525 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:00.209656 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.209590 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret podName:46062402-20c6-4730-8520-b1a94ef3c11a nodeName:}" failed. No retries permitted until 2026-07-06 12:47:02.209571763 +0000 UTC m=+6.150412607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret") pod "global-pull-secret-syncer-gc9tb" (UID: "46062402-20c6-4730-8520-b1a94ef3c11a") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:00.210041 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.210021 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:47:00.210106 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.210051 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:47:00.210106 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.210066 2568 projected.go:196] Error preparing data for projected volume kube-api-access-svn9w for pod openshift-network-diagnostics/network-check-target-w6dgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:00.210204 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.210113 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w podName:f568bda6-dc07-4f9b-9921-32d0dc051147 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:02.210097993 +0000 UTC m=+6.150938845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-svn9w" (UniqueName: "kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w") pod "network-check-target-w6dgr" (UID: "f568bda6-dc07-4f9b-9921-32d0dc051147") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:00.556859 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.556777 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 12:41:58 +0000 UTC" deadline="2028-02-16 17:38:01.197479785 +0000 UTC" Jul 06 12:47:00.556859 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.556816 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14164h51m0.64066776s" Jul 06 12:47:00.585524 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.585491 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:00.585707 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.585610 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:00.588034 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.587996 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:00.588145 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:00.588123 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:00.639149 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.638390 2568 generic.go:356] "Generic (PLEG): container finished" podID="b2b28ef55d482c311313ef51a865f9fa" containerID="3f4c705bad8809b848baca6bd427850d1ef972332942ffd8580067c4d8baab60" exitCode=0 Jul 06 12:47:00.639149 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.638471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" event={"ID":"b2b28ef55d482c311313ef51a865f9fa","Type":"ContainerDied","Data":"3f4c705bad8809b848baca6bd427850d1ef972332942ffd8580067c4d8baab60"} Jul 06 12:47:00.649771 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:00.649739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" event={"ID":"36c64bbcbb7e33685ecee28ec1d660b6","Type":"ContainerStarted","Data":"1c849f7e62fbd454b67306f82a1c3c50e38012d1c778c85347b951349eabbb30"} Jul 06 12:47:01.585491 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:01.585454 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:01.585989 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:01.585593 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:01.679163 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:01.679079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" event={"ID":"b2b28ef55d482c311313ef51a865f9fa","Type":"ContainerStarted","Data":"fdeeede0041e8489e0292e906c29460716e08e872c2784a55fceafc1c4742240"} Jul 06 12:47:01.701222 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:01.700732 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-145.ec2.internal" podStartSLOduration=4.700710498 podStartE2EDuration="4.700710498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:00.67758552 +0000 UTC m=+4.618426380" watchObservedRunningTime="2026-07-06 12:47:01.700710498 +0000 UTC m=+5.641551358" Jul 06 12:47:02.128947 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:02.128370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:02.128947 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.128525 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:02.128947 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.128587 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:06.128568875 +0000 UTC m=+10.069409735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:02.229066 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:02.229024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:02.229232 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:02.229095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:02.229309 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.229277 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:47:02.229309 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.229296 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:47:02.229309 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.229309 2568 projected.go:196] Error preparing data for projected volume kube-api-access-svn9w for pod openshift-network-diagnostics/network-check-target-w6dgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:02.229451 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.229368 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w podName:f568bda6-dc07-4f9b-9921-32d0dc051147 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:06.229349257 +0000 UTC m=+10.170190097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-svn9w" (UniqueName: "kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w") pod "network-check-target-w6dgr" (UID: "f568bda6-dc07-4f9b-9921-32d0dc051147") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:02.229815 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.229793 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:02.229922 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.229854 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret podName:46062402-20c6-4730-8520-b1a94ef3c11a nodeName:}" failed. No retries permitted until 2026-07-06 12:47:06.229838096 +0000 UTC m=+10.170678939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret") pod "global-pull-secret-syncer-gc9tb" (UID: "46062402-20c6-4730-8520-b1a94ef3c11a") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:02.585578 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:02.584816 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:02.585578 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.584956 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:02.585578 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:02.585298 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:02.585578 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:02.585372 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:03.584892 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:03.584856 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:03.585126 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:03.584989 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:04.587212 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:04.587180 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:04.587212 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:04.587192 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:04.587723 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:04.587294 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:04.587723 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:04.587352 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:05.585180 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:05.584740 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:05.585180 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:05.584879 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:06.165163 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:06.165120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:06.165653 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.165309 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:06.165653 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.165374 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:14.165354008 +0000 UTC m=+18.106194847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:06.266180 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:06.266141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:06.266342 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:06.266201 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:06.266422 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.266356 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:47:06.266422 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.266374 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:47:06.266422 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.266386 2568 projected.go:196] Error preparing data for projected volume kube-api-access-svn9w for pod openshift-network-diagnostics/network-check-target-w6dgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:06.266547 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.266444 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w podName:f568bda6-dc07-4f9b-9921-32d0dc051147 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:14.266426283 +0000 UTC m=+18.207267138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-svn9w" (UniqueName: "kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w") pod "network-check-target-w6dgr" (UID: "f568bda6-dc07-4f9b-9921-32d0dc051147") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:06.266855 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.266835 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:06.266972 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.266888 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret podName:46062402-20c6-4730-8520-b1a94ef3c11a nodeName:}" failed. No retries permitted until 2026-07-06 12:47:14.266873836 +0000 UTC m=+18.207714689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret") pod "global-pull-secret-syncer-gc9tb" (UID: "46062402-20c6-4730-8520-b1a94ef3c11a") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:06.586810 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:06.586733 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:06.586958 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.586850 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:06.587251 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:06.587225 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:06.587380 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:06.587334 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:07.585349 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:07.585319 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:07.585810 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:07.585455 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:08.585611 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:08.585577 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:08.586026 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:08.585714 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:08.586332 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:08.586310 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:08.586422 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:08.586404 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:09.585207 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:09.585174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:09.585398 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:09.585291 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:10.585356 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:10.585325 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:10.585751 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:10.585460 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:10.585751 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:10.585540 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:10.585751 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:10.585674 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:11.585062 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:11.585028 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:11.585234 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:11.585162 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:12.585171 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:12.585135 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:12.585617 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:12.585133 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:12.585728 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:12.585671 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:12.586273 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:12.586222 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:13.585124 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:13.585084 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:13.585402 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:13.585208 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:14.228146 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:14.228125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:14.228255 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.228243 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:14.228300 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.228292 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:30.228279628 +0000 UTC m=+34.169120465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:14.328681 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:14.328653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:14.328832 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:14.328696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:14.328832 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.328785 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:14.328832 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.328790 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:47:14.328968 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.328850 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret podName:46062402-20c6-4730-8520-b1a94ef3c11a nodeName:}" failed. No retries permitted until 2026-07-06 12:47:30.328831606 +0000 UTC m=+34.269672458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret") pod "global-pull-secret-syncer-gc9tb" (UID: "46062402-20c6-4730-8520-b1a94ef3c11a") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:14.328968 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.328856 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:47:14.328968 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.328870 2568 projected.go:196] Error preparing data for projected volume kube-api-access-svn9w for pod openshift-network-diagnostics/network-check-target-w6dgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:14.328968 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.328917 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w podName:f568bda6-dc07-4f9b-9921-32d0dc051147 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:30.32890565 +0000 UTC m=+34.269746504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-svn9w" (UniqueName: "kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w") pod "network-check-target-w6dgr" (UID: "f568bda6-dc07-4f9b-9921-32d0dc051147") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:14.585305 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:14.585226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:14.585459 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.585363 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:14.585459 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:14.585419 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:14.585909 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:14.585523 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:15.584996 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:15.584963 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:15.585183 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:15.585090 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:16.585401 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:16.585373 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:16.585871 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:16.585503 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:16.585871 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:16.585580 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:16.585871 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:16.585694 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:17.585015 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:17.584984 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:17.585188 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:17.585091 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:18.584557 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:18.584532 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:18.584557 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:18.584547 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:18.584930 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:18.584657 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:18.584930 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:18.584735 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:18.716652 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:18.716416 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r7g6r" event={"ID":"db4973a1-3cc2-4a94-aa2b-18d04d82a7d9","Type":"ContainerStarted","Data":"1b02e0795b2043c7a70f34d52d8cfafddcdb273a83bb016eafe149ee3f4329e2"} Jul 06 12:47:18.745136 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:18.745103 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-145.ec2.internal" podStartSLOduration=21.745063376 podStartE2EDuration="21.745063376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:01.700980278 +0000 UTC m=+5.641821138" watchObservedRunningTime="2026-07-06 12:47:18.745063376 +0000 UTC m=+22.685904233" Jul 06 12:47:18.746195 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:18.746070 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r7g6r" podStartSLOduration=8.102498507 podStartE2EDuration="22.746050721s" totalImagesPullingTime="14.643552214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:18.744966137 +0000 UTC m=+22.685806997" watchObservedRunningTime="2026-07-06 12:47:18.746050721 +0000 UTC m=+22.686891581" Jul 06 12:47:18.805431 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:18.805405 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:47:18.806445 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:18.806416 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:47:19.585043 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.584884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:19.585748 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:19.585136 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:19.722112 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.722074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g86l7" event={"ID":"18cc5d81-f939-4834-b1f8-80f4d2e090ce","Type":"ContainerStarted","Data":"6a7b9951686bbd894b9f32e6150b1d8eb4342b92d6bb8973f42f4517f2e850f5"} Jul 06 12:47:19.723880 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.723843 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbb7x" event={"ID":"d2168b17-6206-410a-a930-f665ddda267a","Type":"ContainerStarted","Data":"7cbbe858c5b9358bb6f53f5113f91480a1087730e685e5647408e7c9620e09b6"} Jul 06 12:47:19.725401 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.725375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mxk6l" event={"ID":"6e9f0cfa-f895-4d16-b74d-1cb8d6877cbb","Type":"ContainerStarted","Data":"b814679b54b329108976e0f46b32c769e8fbf902476ed46c555aa8b48296c675"} Jul 06 12:47:19.726924 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.726893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" event={"ID":"1af5ef28-47db-42d8-8bd0-24b289863efc","Type":"ContainerStarted","Data":"537a439b131143bb82b7a233d73c34ad4ebbf35980a96ee66ac9789a8e08acab"} Jul 06 12:47:19.728465 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.728441 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9x5md" event={"ID":"efdb6ed1-7693-4154-8cd4-94005582afff","Type":"ContainerStarted","Data":"6b7418b3cfbece9fa56e7928c80c9d9cb987985838117205fb7a7d2664961a04"} Jul 06 12:47:19.731483 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.731463 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:47:19.731843 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.731814 2568 generic.go:356] "Generic (PLEG): container finished" podID="f46d4222-907b-4cb6-8217-38e8228ee9d6" containerID="4f88dbdaf99be17121d72f59ea014f99d4bb551ab260d46ea92302206749690d" exitCode=1 Jul 06 12:47:19.731932 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.731878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"7cb65f31b12523205db487cede6bb08c776fbd7e0fbc233b913ee4d0ec94cd63"} Jul 06 12:47:19.731977 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.731943 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"274e4937f13679e3e44f674b98585b0bd8f17a19bb6be1f038385e515ea06fdc"} Jul 06 12:47:19.732027 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.731998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"36c46f68e2b24ffea72ab22ebc2d2f312f88d98ddb0011673de7cf63b01ffacd"} Jul 06 12:47:19.732066 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.732048 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"97b37fff1c4d267a35fc7f4ffe3785f3329c7dc3bccd14c5187620089d682a99"} Jul 06 12:47:19.732112 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.732095 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerDied","Data":"4f88dbdaf99be17121d72f59ea014f99d4bb551ab260d46ea92302206749690d"} Jul 06 12:47:19.732163 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.732143 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"85f174cbd4438d9af00b9373357d59c70d6829b96a52224e566ac09a34978953"} Jul 06 12:47:19.733396 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.733375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" event={"ID":"76e28400-b5b5-423d-9a3c-f4731f5e8121","Type":"ContainerStarted","Data":"054e651ad83bd5dec1a53e80be50987c2c172ce85f544c9b10cbc27f6c63c220"} Jul 06 12:47:19.735090 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.735067 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d398316-6b06-4e5e-8e3d-174be9a72a8b" containerID="dd1c568ee6d1a0fc21e0c0a19b967630d6a029e6f40e325070eca4bf6aa97a56" exitCode=0 Jul 06 12:47:19.735198 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.735145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerDied","Data":"dd1c568ee6d1a0fc21e0c0a19b967630d6a029e6f40e325070eca4bf6aa97a56"} Jul 06 12:47:19.739724 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.739676 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g86l7" podStartSLOduration=4.805306755 podStartE2EDuration="23.739659637s" totalImagesPullingTime="18.934352882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:19.739346389 +0000 UTC m=+23.680187242" watchObservedRunningTime="2026-07-06 12:47:19.739659637 +0000 UTC m=+23.680500496" Jul 06 12:47:19.787982 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.787932 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mxk6l" podStartSLOduration=4.785589403 podStartE2EDuration="23.787917697s" totalImagesPullingTime="19.002328294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:19.787859975 +0000 UTC m=+23.728700831" watchObservedRunningTime="2026-07-06 12:47:19.787917697 +0000 UTC m=+23.728758555" Jul 06 12:47:19.788158 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.788137 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbb7x" podStartSLOduration=4.832038828 podStartE2EDuration="23.788132803s" totalImagesPullingTime="18.956093975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:19.760200561 +0000 UTC m=+23.701041420" watchObservedRunningTime="2026-07-06 12:47:19.788132803 +0000 UTC m=+23.728973660" Jul 06 12:47:19.808726 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.808666 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rg5g8" podStartSLOduration=4.847387756 podStartE2EDuration="23.80862707s" totalImagesPullingTime="18.961239314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:19.807856441 +0000 UTC m=+23.748697280" watchObservedRunningTime="2026-07-06 12:47:19.80862707 +0000 UTC m=+23.749467930" Jul 06 12:47:19.830621 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.830558 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9x5md" podStartSLOduration=4.887546882 podStartE2EDuration="23.83054592s" totalImagesPullingTime="18.942999038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:19.830389313 +0000 UTC m=+23.771230182" watchObservedRunningTime="2026-07-06 12:47:19.83054592 +0000 UTC m=+23.771386779" Jul 06 12:47:19.953707 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:19.953676 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:20.054169 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.054133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:20.224684 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.224664 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 12:47:20.235305 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.235279 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:20.336698 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.336582 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:20.553481 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.553392 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T12:47:20.224678891Z","UUID":"d49658d9-4a34-4b47-a03b-213b231f831a","Handler":null,"Name":"","Endpoint":""} Jul 06 12:47:20.555161 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.555135 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 12:47:20.555291 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.555180 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 12:47:20.585429 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.585406 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:20.585850 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.585418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:20.585850 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:20.585525 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:20.585850 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:20.585648 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:20.739706 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.739671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" event={"ID":"1af5ef28-47db-42d8-8bd0-24b289863efc","Type":"ContainerStarted","Data":"b4c2092001f015ab9c29684436e1adc27e37250eee132d2ab4aa9b812b7f1a8c"} Jul 06 12:47:20.740335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.740314 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 12:47:20.870768 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.870720 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:20.977393 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:20.977360 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:21.218359 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:21.218305 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:21.312424 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:21.312391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:21.585325 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:21.585113 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:21.585511 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:21.585361 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:21.745181 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:21.745153 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:47:21.745746 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:21.745721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"af3c95b6b1347e43983ed52819145e1c4007a91c2142f242dc5a1f48ae79ce1c"} Jul 06 12:47:21.749801 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:21.749773 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" event={"ID":"1af5ef28-47db-42d8-8bd0-24b289863efc","Type":"ContainerStarted","Data":"a872e208d21005c003fe8ec3a3698df550be25fb63b0c02f1b41846cf5d16008"} Jul 06 12:47:21.769275 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:21.769226 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kthn2" podStartSLOduration=4.523479953 podStartE2EDuration="25.769213104s" totalImagesPullingTime="21.245733151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:21.76891174 +0000 UTC m=+25.709752600" watchObservedRunningTime="2026-07-06 12:47:21.769213104 +0000 UTC m=+25.710053963" Jul 06 12:47:22.585294 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:22.585261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:22.585482 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:22.585277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:22.585482 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:22.585408 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:22.585482 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:22.585459 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:23.584651 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.584606 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:23.584787 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:23.584766 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:23.758259 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.758174 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:47:23.759393 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.759191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"859ce36ac77971728f587f356ecd39d40a4646769f3e87511c390e0b2c51891f"} Jul 06 12:47:23.759842 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.759815 2568 scope.go:117] "RemoveContainer" containerID="4f88dbdaf99be17121d72f59ea014f99d4bb551ab260d46ea92302206749690d" Jul 06 12:47:23.759965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.759889 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:47:23.760024 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.759967 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:47:23.760024 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.760015 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:47:23.776260 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.776239 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:47:23.776497 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:23.776483 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:47:24.584763 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:24.584729 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:24.585040 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:24.584729 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:24.585040 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:24.584876 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:24.585040 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:24.584933 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:25.428675 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.428646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w6dgr"] Jul 06 12:47:25.429076 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.428927 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:25.429076 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:25.429032 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:25.431610 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.431583 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8n477"] Jul 06 12:47:25.431733 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.431698 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:25.431855 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:25.431822 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:25.441064 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.441039 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gc9tb"] Jul 06 12:47:25.441162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.441148 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:25.441264 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:25.441242 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:25.768020 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.767960 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:47:25.768439 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.768413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" event={"ID":"f46d4222-907b-4cb6-8217-38e8228ee9d6","Type":"ContainerStarted","Data":"dd8ff50dd42792c79d32d716dfd9cd4944e77682c2f09d9788f88467e32d43fb"} Jul 06 12:47:25.770499 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.770472 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d398316-6b06-4e5e-8e3d-174be9a72a8b" containerID="a0e61c9dba87e1f6da01747f93ce7264985a01d5c294fc7b863017b31368a5b3" exitCode=0 Jul 06 12:47:25.770603 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.770530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerDied","Data":"a0e61c9dba87e1f6da01747f93ce7264985a01d5c294fc7b863017b31368a5b3"} Jul 06 12:47:25.797290 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:25.797249 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" podStartSLOduration=10.813920085 podStartE2EDuration="29.797235499s" totalImagesPullingTime="18.983315414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:25.797211214 +0000 UTC m=+29.738052098" watchObservedRunningTime="2026-07-06 12:47:25.797235499 +0000 UTC m=+29.738076352" Jul 06 12:47:26.585350 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:26.585182 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:26.585725 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:26.585247 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:26.585725 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:26.585422 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:26.585725 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:26.585533 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:26.828909 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:26.828878 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:47:26.829075 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:26.829011 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 12:47:26.829459 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:26.829436 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r7g6r" Jul 06 12:47:26.918604 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:26.918565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:27.240156 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:27.240097 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:27.341193 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:27.341163 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" limit=1024 Jul 06 12:47:27.585332 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:27.585307 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:27.585480 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:27.585398 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:27.777413 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:27.777335 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d398316-6b06-4e5e-8e3d-174be9a72a8b" containerID="43b8b1798c339481750624848cbcd6457e60d62a48a70ea6413da99e2f32fafc" exitCode=0 Jul 06 12:47:27.777554 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:27.777457 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerDied","Data":"43b8b1798c339481750624848cbcd6457e60d62a48a70ea6413da99e2f32fafc"} Jul 06 12:47:28.585187 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:28.585155 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:28.585301 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:28.585163 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:28.585357 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:28.585307 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:28.585357 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:28.585337 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:28.783907 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:28.783842 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d398316-6b06-4e5e-8e3d-174be9a72a8b" containerID="cb3cc96f18af2e6db03752ddf64b842e79e74019ca267e200f82b68f06a26326" exitCode=0 Jul 06 12:47:28.783907 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:28.783884 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerDied","Data":"cb3cc96f18af2e6db03752ddf64b842e79e74019ca267e200f82b68f06a26326"} Jul 06 12:47:29.584497 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:29.584469 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:29.584666 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:29.584570 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6dgr" podUID="f568bda6-dc07-4f9b-9921-32d0dc051147" Jul 06 12:47:30.254770 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:30.254731 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:30.255233 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.254872 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:30.255233 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.254949 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:02.254929528 +0000 UTC m=+66.195770368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 12:47:30.355759 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:30.355726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:30.355942 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:30.355798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:30.355942 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.355901 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:30.355942 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.355912 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 12:47:30.355942 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.355938 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 12:47:30.356146 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.355951 2568 projected.go:196] Error preparing data for projected volume kube-api-access-svn9w for pod openshift-network-diagnostics/network-check-target-w6dgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:30.356146 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.355964 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret podName:46062402-20c6-4730-8520-b1a94ef3c11a nodeName:}" failed. No retries permitted until 2026-07-06 12:48:02.355947201 +0000 UTC m=+66.296788055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret") pod "global-pull-secret-syncer-gc9tb" (UID: "46062402-20c6-4730-8520-b1a94ef3c11a") : object "kube-system"/"original-pull-secret" not registered Jul 06 12:47:30.356146 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.355990 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w podName:f568bda6-dc07-4f9b-9921-32d0dc051147 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:02.355979603 +0000 UTC m=+66.296820440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-svn9w" (UniqueName: "kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w") pod "network-check-target-w6dgr" (UID: "f568bda6-dc07-4f9b-9921-32d0dc051147") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 12:47:30.584611 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:30.584529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:30.584777 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.584665 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gc9tb" podUID="46062402-20c6-4730-8520-b1a94ef3c11a" Jul 06 12:47:30.584777 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:30.584741 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:30.584915 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:30.584891 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8n477" podUID="d42744ee-0140-41b4-afba-64fd0483ee84" Jul 06 12:47:31.337623 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.337586 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-145.ec2.internal" event="NodeReady" Jul 06 12:47:31.338066 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.337772 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 12:47:31.398958 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.398934 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m8fp5"] Jul 06 12:47:31.401028 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.401004 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fvt4q"] Jul 06 12:47:31.401152 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.401136 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.402842 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.402820 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.406901 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.405045 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 12:47:31.406901 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.405263 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 12:47:31.406901 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.405330 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 12:47:31.406901 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.405606 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-28thq\"" Jul 06 12:47:31.406901 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.406030 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9njlh\"" Jul 06 12:47:31.406901 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.406275 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 12:47:31.406901 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.406375 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 12:47:31.416522 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.416499 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8fp5"] Jul 06 12:47:31.418225 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.418202 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fvt4q"] Jul 06 12:47:31.566593 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.566511 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-config-volume\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.566593 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.566555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-tmp-dir\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.566593 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.566583 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9r86\" (UniqueName: \"kubernetes.io/projected/77930e36-25bd-46b1-9808-8c87f40fc07e-kube-api-access-t9r86\") pod \"ingress-canary-m8fp5\" (UID: \"77930e36-25bd-46b1-9808-8c87f40fc07e\") " pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.566810 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.566688 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xsv\" (UniqueName: \"kubernetes.io/projected/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-kube-api-access-l8xsv\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.566810 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.566754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.566810 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.566778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77930e36-25bd-46b1-9808-8c87f40fc07e-cert\") pod \"ingress-canary-m8fp5\" (UID: \"77930e36-25bd-46b1-9808-8c87f40fc07e\") " pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.584894 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.584873 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:47:31.587509 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.587491 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 12:47:31.587797 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.587783 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mmbsd\"" Jul 06 12:47:31.587879 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.587862 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 12:47:31.667894 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.667867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.668021 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.667899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77930e36-25bd-46b1-9808-8c87f40fc07e-cert\") pod \"ingress-canary-m8fp5\" (UID: \"77930e36-25bd-46b1-9808-8c87f40fc07e\") " pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.668021 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.667930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-config-volume\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.668021 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.667971 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-tmp-dir\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.668021 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.667995 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9r86\" (UniqueName: \"kubernetes.io/projected/77930e36-25bd-46b1-9808-8c87f40fc07e-kube-api-access-t9r86\") pod \"ingress-canary-m8fp5\" (UID: \"77930e36-25bd-46b1-9808-8c87f40fc07e\") " pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.668021 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:31.668016 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:47:31.668253 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:31.668085 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:32.168064211 +0000 UTC m=+36.108905060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:47:31.668253 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.668235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xsv\" (UniqueName: \"kubernetes.io/projected/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-kube-api-access-l8xsv\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.668360 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.668336 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-tmp-dir\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.668676 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.668658 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-config-volume\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.672621 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.672602 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77930e36-25bd-46b1-9808-8c87f40fc07e-cert\") pod \"ingress-canary-m8fp5\" (UID: \"77930e36-25bd-46b1-9808-8c87f40fc07e\") " pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.677789 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.677568 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xsv\" (UniqueName: \"kubernetes.io/projected/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-kube-api-access-l8xsv\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:31.677907 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.677890 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9r86\" (UniqueName: \"kubernetes.io/projected/77930e36-25bd-46b1-9808-8c87f40fc07e-kube-api-access-t9r86\") pod \"ingress-canary-m8fp5\" (UID: \"77930e36-25bd-46b1-9808-8c87f40fc07e\") " pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.716625 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.716599 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8fp5" Jul 06 12:47:31.867671 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:31.867622 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8fp5"] Jul 06 12:47:32.172175 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:32.172077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:32.172369 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:32.172261 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:47:32.172369 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:32.172338 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:33.17231798 +0000 UTC m=+37.113158837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:47:32.585363 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:32.585280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:47:32.585820 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:32.585526 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:47:32.588525 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:32.588501 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dxdc8\"" Jul 06 12:47:32.588663 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:32.588502 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 12:47:32.589440 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:32.589416 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 12:47:33.177337 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:33.177291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:33.177517 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:33.177448 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:47:33.177517 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:33.177507 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:35.17748974 +0000 UTC m=+39.118330588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:47:33.428250 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:47:33.428181 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77930e36_25bd_46b1_9808_8c87f40fc07e.slice/crio-dfd80c00b84212ef2e6ff69ccc71b8e707e55eb9fbac98e0cf3a9f11ffed2966 WatchSource:0}: Error finding container dfd80c00b84212ef2e6ff69ccc71b8e707e55eb9fbac98e0cf3a9f11ffed2966: Status 404 returned error can't find the container with id dfd80c00b84212ef2e6ff69ccc71b8e707e55eb9fbac98e0cf3a9f11ffed2966 Jul 06 12:47:33.796762 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:33.796733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8fp5" event={"ID":"77930e36-25bd-46b1-9808-8c87f40fc07e","Type":"ContainerStarted","Data":"dfd80c00b84212ef2e6ff69ccc71b8e707e55eb9fbac98e0cf3a9f11ffed2966"} Jul 06 12:47:34.802790 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:34.802760 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d398316-6b06-4e5e-8e3d-174be9a72a8b" containerID="acab0b1cdcf3ed102e3e349fec8e93568c47abcd67184d417b40daafba347c85" exitCode=0 Jul 06 12:47:34.803244 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:34.802798 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerDied","Data":"acab0b1cdcf3ed102e3e349fec8e93568c47abcd67184d417b40daafba347c85"} Jul 06 12:47:35.191256 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:35.191094 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:35.191397 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:35.191270 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:47:35.191397 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:35.191393 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:39.191369284 +0000 UTC m=+43.132210135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:47:35.809512 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:35.809475 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d398316-6b06-4e5e-8e3d-174be9a72a8b" containerID="f42a3e737a7436d5fcf0197364a0bdbeb052515eda6d6c6cb38410069bc8157f" exitCode=0 Jul 06 12:47:35.809942 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:35.809536 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerDied","Data":"f42a3e737a7436d5fcf0197364a0bdbeb052515eda6d6c6cb38410069bc8157f"} Jul 06 12:47:36.815482 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:36.815443 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" event={"ID":"2d398316-6b06-4e5e-8e3d-174be9a72a8b","Type":"ContainerStarted","Data":"aeb389508ac0a49874ae7d99ef851804e6740b7b976d3555127be0ebac690b64"} Jul 06 12:47:36.816814 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:36.816788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8fp5" event={"ID":"77930e36-25bd-46b1-9808-8c87f40fc07e","Type":"ContainerStarted","Data":"90fb5aec6e77318096f1f7bf7df611f68ee2dc24e8c933957e5d9eef80254292"} Jul 06 12:47:36.853950 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:36.853893 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9jnkb" podStartSLOduration=9.196525486 podStartE2EDuration="40.853873458s" totalImagesPullingTime="31.657347972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:36.850688108 +0000 UTC m=+40.791528967" watchObservedRunningTime="2026-07-06 12:47:36.853873458 +0000 UTC m=+40.794714319" Jul 06 12:47:39.222205 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:39.222166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:39.222602 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:39.222302 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:47:39.222602 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:39.222358 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:47:47.22234272 +0000 UTC m=+51.163183561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:47:47.272623 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:47.272238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:47:47.273337 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:47.272699 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:47:47.273502 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:47:47.273490 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:03.273466873 +0000 UTC m=+67.214307724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:47:56.783383 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:56.783353 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cmpd" Jul 06 12:47:56.855147 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:47:56.855099 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m8fp5" podStartSLOduration=22.663829361 podStartE2EDuration="25.855084353s" totalImagesPullingTime="3.191254992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:47:36.877762077 +0000 UTC m=+40.818602936" watchObservedRunningTime="2026-07-06 12:47:56.855084353 +0000 UTC m=+60.795925220" Jul 06 12:48:02.273921 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.273883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:48:02.280355 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.280330 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 12:48:02.284911 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:02.284892 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 12:48:02.284989 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:02.284973 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs podName:d42744ee-0140-41b4-afba-64fd0483ee84 nodeName:}" failed. No retries permitted until 2026-07-06 12:49:06.284954529 +0000 UTC m=+130.225795366 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs") pod "network-metrics-daemon-8n477" (UID: "d42744ee-0140-41b4-afba-64fd0483ee84") : secret "metrics-daemon-secret" not found Jul 06 12:48:02.374942 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.374909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:48:02.375029 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.374953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:48:02.378162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.378143 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 12:48:02.378690 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.378673 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 12:48:02.388307 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.388283 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 12:48:02.389387 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.389365 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46062402-20c6-4730-8520-b1a94ef3c11a-original-pull-secret\") pod \"global-pull-secret-syncer-gc9tb\" (UID: \"46062402-20c6-4730-8520-b1a94ef3c11a\") " pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:48:02.398734 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.398709 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svn9w\" (UniqueName: \"kubernetes.io/projected/f568bda6-dc07-4f9b-9921-32d0dc051147-kube-api-access-svn9w\") pod \"network-check-target-w6dgr\" (UID: \"f568bda6-dc07-4f9b-9921-32d0dc051147\") " pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:48:02.495978 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.495949 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mmbsd\"" Jul 06 12:48:02.504061 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.504045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:48:02.596435 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.596409 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gc9tb" Jul 06 12:48:02.638187 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.638123 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w6dgr"] Jul 06 12:48:02.642455 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:02.642428 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf568bda6_dc07_4f9b_9921_32d0dc051147.slice/crio-06216851d4381bb1dc57a93c929c4858d9307d8cd5dc0a1b9460fdd02426220d WatchSource:0}: Error finding container 06216851d4381bb1dc57a93c929c4858d9307d8cd5dc0a1b9460fdd02426220d: Status 404 returned error can't find the container with id 06216851d4381bb1dc57a93c929c4858d9307d8cd5dc0a1b9460fdd02426220d Jul 06 12:48:02.728998 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.728970 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gc9tb"] Jul 06 12:48:02.732489 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:02.732460 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46062402_20c6_4730_8520_b1a94ef3c11a.slice/crio-04eed5a99160e5816148f5b88b29df33080a86dddb257c40e723302b7697826e WatchSource:0}: Error finding container 04eed5a99160e5816148f5b88b29df33080a86dddb257c40e723302b7697826e: Status 404 returned error can't find the container with id 04eed5a99160e5816148f5b88b29df33080a86dddb257c40e723302b7697826e Jul 06 12:48:02.876370 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.876333 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gc9tb" event={"ID":"46062402-20c6-4730-8520-b1a94ef3c11a","Type":"ContainerStarted","Data":"04eed5a99160e5816148f5b88b29df33080a86dddb257c40e723302b7697826e"} Jul 06 12:48:02.877441 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:02.877415 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w6dgr" event={"ID":"f568bda6-dc07-4f9b-9921-32d0dc051147","Type":"ContainerStarted","Data":"06216851d4381bb1dc57a93c929c4858d9307d8cd5dc0a1b9460fdd02426220d"} Jul 06 12:48:03.282992 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:03.282909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:48:03.283334 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:03.283040 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:48:03.283334 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:03.283092 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:35.283079525 +0000 UTC m=+99.223920362 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:48:08.893880 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:08.893839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gc9tb" event={"ID":"46062402-20c6-4730-8520-b1a94ef3c11a","Type":"ContainerStarted","Data":"04f8869bfdf629344129334af749ab14c993d31bcb79a19188cc2c22d2e47927"} Jul 06 12:48:08.895322 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:08.895301 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w6dgr" event={"ID":"f568bda6-dc07-4f9b-9921-32d0dc051147","Type":"ContainerStarted","Data":"8a62faafc3ae8dc5cc94221899e115684944013af957c6a3a35fb28ae9394225"} Jul 06 12:48:08.895453 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:08.895434 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:48:08.920380 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:08.918479 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gc9tb" podStartSLOduration=67.739037946 podStartE2EDuration="1m12.918461199s" totalImagesPullingTime="5.179423253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:08.91700304 +0000 UTC m=+72.857843894" watchObservedRunningTime="2026-07-06 12:48:08.918461199 +0000 UTC m=+72.859302062" Jul 06 12:48:08.937893 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:08.937853 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w6dgr" podStartSLOduration=67.67924827 podStartE2EDuration="1m12.937839707s" totalImagesPullingTime="5.258591437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:08.937455059 +0000 UTC m=+72.878295934" watchObservedRunningTime="2026-07-06 12:48:08.937839707 +0000 UTC m=+72.878680593" Jul 06 12:48:23.508304 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.508147 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xnzpc"] Jul 06 12:48:23.512314 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.512299 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.515179 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.515162 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 12:48:23.515935 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.515917 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 12:48:23.516019 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.515939 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 12:48:23.516019 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.515939 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5zxp7\"" Jul 06 12:48:23.516274 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.516260 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 12:48:23.521676 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.521657 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 12:48:23.522846 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.522824 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xnzpc"] Jul 06 12:48:23.614069 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.614044 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4"] Jul 06 12:48:23.614205 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.614186 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec35516-c4f2-40e3-bbae-fb91296ddf06-serving-cert\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.614256 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.614220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pxq\" (UniqueName: \"kubernetes.io/projected/3ec35516-c4f2-40e3-bbae-fb91296ddf06-kube-api-access-97pxq\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.614256 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.614246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec35516-c4f2-40e3-bbae-fb91296ddf06-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.614365 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.614292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec35516-c4f2-40e3-bbae-fb91296ddf06-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.614365 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.614318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec35516-c4f2-40e3-bbae-fb91296ddf06-tmp\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.614365 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.614352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ec35516-c4f2-40e3-bbae-fb91296ddf06-snapshots\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.616943 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.616929 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:23.620515 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.620500 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 12:48:23.620597 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.620556 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 12:48:23.620776 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.620763 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fnvpq\"" Jul 06 12:48:23.620875 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.620859 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:48:23.629465 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.629447 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4"] Jul 06 12:48:23.715320 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhmv\" (UniqueName: \"kubernetes.io/projected/0fef59b9-4f17-45f7-8fee-a14b48278706-kube-api-access-7rhmv\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:23.715423 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec35516-c4f2-40e3-bbae-fb91296ddf06-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.715423 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:23.715423 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec35516-c4f2-40e3-bbae-fb91296ddf06-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.715530 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec35516-c4f2-40e3-bbae-fb91296ddf06-tmp\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.715530 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ec35516-c4f2-40e3-bbae-fb91296ddf06-snapshots\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.715530 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec35516-c4f2-40e3-bbae-fb91296ddf06-serving-cert\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.715655 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pxq\" (UniqueName: \"kubernetes.io/projected/3ec35516-c4f2-40e3-bbae-fb91296ddf06-kube-api-access-97pxq\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.715904 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715885 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec35516-c4f2-40e3-bbae-fb91296ddf06-tmp\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.715983 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.715960 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec35516-c4f2-40e3-bbae-fb91296ddf06-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.716141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.716122 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ec35516-c4f2-40e3-bbae-fb91296ddf06-snapshots\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.716352 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.716335 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec35516-c4f2-40e3-bbae-fb91296ddf06-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.718032 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.718015 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec35516-c4f2-40e3-bbae-fb91296ddf06-serving-cert\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.724112 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.724090 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fc7fc97d5-fgw7g"] Jul 06 12:48:23.727013 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.726999 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.736556 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.736537 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5"] Jul 06 12:48:23.739096 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.739073 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" Jul 06 12:48:23.749992 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.749963 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 12:48:23.750204 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.750185 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 12:48:23.750311 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.750190 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 12:48:23.750375 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.750343 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:48:23.750474 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.750457 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drj84\"" Jul 06 12:48:23.750538 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.750524 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 12:48:23.750955 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.750938 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rh94s\"" Jul 06 12:48:23.752861 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.752844 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pxq\" (UniqueName: \"kubernetes.io/projected/3ec35516-c4f2-40e3-bbae-fb91296ddf06-kube-api-access-97pxq\") pod \"insights-operator-5c847dcfc4-xnzpc\" (UID: \"3ec35516-c4f2-40e3-bbae-fb91296ddf06\") " pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.758409 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.758369 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 12:48:23.769174 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.769149 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc7fc97d5-fgw7g"] Jul 06 12:48:23.776204 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.776183 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5"] Jul 06 12:48:23.816843 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.816815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhmv\" (UniqueName: \"kubernetes.io/projected/0fef59b9-4f17-45f7-8fee-a14b48278706-kube-api-access-7rhmv\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:23.816949 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.816845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-image-registry-private-configuration\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.816949 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.816874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:23.816949 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.816890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-installation-pull-secrets\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.816949 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.816909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-ca-trust-extracted\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.817093 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.816954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5pbh\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-kube-api-access-b5pbh\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.817093 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.817005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.817093 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:23.817015 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 12:48:23.817093 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.817029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-bound-sa-token\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.817093 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:23.817081 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls podName:0fef59b9-4f17-45f7-8fee-a14b48278706 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:24.317062485 +0000 UTC m=+88.257903337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-fz2l4" (UID: "0fef59b9-4f17-45f7-8fee-a14b48278706") : secret "samples-operator-tls" not found Jul 06 12:48:23.817306 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.817098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddv5\" (UniqueName: \"kubernetes.io/projected/826fcfd6-3c84-4370-a599-5862b66e91f5-kube-api-access-gddv5\") pod \"volume-data-source-validator-5d456fdbcd-l5mj5\" (UID: \"826fcfd6-3c84-4370-a599-5862b66e91f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" Jul 06 12:48:23.817306 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.817119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-certificates\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.817306 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.817137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-trusted-ca\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.820805 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.820788 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" Jul 06 12:48:23.828293 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.828274 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhmv\" (UniqueName: \"kubernetes.io/projected/0fef59b9-4f17-45f7-8fee-a14b48278706-kube-api-access-7rhmv\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:23.918351 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-bound-sa-token\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.918471 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddv5\" (UniqueName: \"kubernetes.io/projected/826fcfd6-3c84-4370-a599-5862b66e91f5-kube-api-access-gddv5\") pod \"volume-data-source-validator-5d456fdbcd-l5mj5\" (UID: \"826fcfd6-3c84-4370-a599-5862b66e91f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" Jul 06 12:48:23.918471 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-certificates\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.918471 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918457 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-trusted-ca\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.918582 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918536 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-image-registry-private-configuration\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.918618 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-installation-pull-secrets\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.918687 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-ca-trust-extracted\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.918860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5pbh\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-kube-api-access-b5pbh\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.918927 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.918894 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.919260 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:23.919000 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:48:23.919260 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:23.919018 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc7fc97d5-fgw7g: secret "image-registry-tls" not found Jul 06 12:48:23.919260 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:23.919082 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls podName:d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:24.419065373 +0000 UTC m=+88.359906227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls") pod "image-registry-7fc7fc97d5-fgw7g" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5") : secret "image-registry-tls" not found Jul 06 12:48:23.919260 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.919155 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-certificates\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.919260 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.919195 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-ca-trust-extracted\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.919756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.919653 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-trusted-ca\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.921237 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.921217 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-image-registry-private-configuration\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.921480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.921459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-installation-pull-secrets\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.928298 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.928276 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5pbh\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-kube-api-access-b5pbh\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.928395 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.928345 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-bound-sa-token\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:23.928438 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.928418 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddv5\" (UniqueName: \"kubernetes.io/projected/826fcfd6-3c84-4370-a599-5862b66e91f5-kube-api-access-gddv5\") pod \"volume-data-source-validator-5d456fdbcd-l5mj5\" (UID: \"826fcfd6-3c84-4370-a599-5862b66e91f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" Jul 06 12:48:23.950431 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:23.950410 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xnzpc"] Jul 06 12:48:23.953167 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:23.953147 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec35516_c4f2_40e3_bbae_fb91296ddf06.slice/crio-6c07cc410800758e6a2e99268fd84b3e26e6bfa2e06b54dc538bbfa46c0a49b7 WatchSource:0}: Error finding container 6c07cc410800758e6a2e99268fd84b3e26e6bfa2e06b54dc538bbfa46c0a49b7: Status 404 returned error can't find the container with id 6c07cc410800758e6a2e99268fd84b3e26e6bfa2e06b54dc538bbfa46c0a49b7 Jul 06 12:48:24.046919 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:24.046864 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" Jul 06 12:48:24.168445 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:24.168416 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5"] Jul 06 12:48:24.171155 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:24.171128 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826fcfd6_3c84_4370_a599_5862b66e91f5.slice/crio-ddf6396686259261e04af870d5ad866cbf658f25c851aa7f6a853aa86de7b52b WatchSource:0}: Error finding container ddf6396686259261e04af870d5ad866cbf658f25c851aa7f6a853aa86de7b52b: Status 404 returned error can't find the container with id ddf6396686259261e04af870d5ad866cbf658f25c851aa7f6a853aa86de7b52b Jul 06 12:48:24.322735 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:24.322588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:24.322735 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:24.322686 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 12:48:24.322904 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:24.322745 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls podName:0fef59b9-4f17-45f7-8fee-a14b48278706 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:25.322731468 +0000 UTC m=+89.263572305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-fz2l4" (UID: "0fef59b9-4f17-45f7-8fee-a14b48278706") : secret "samples-operator-tls" not found Jul 06 12:48:24.424120 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:24.424091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:24.424284 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:24.424264 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:48:24.424336 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:24.424289 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc7fc97d5-fgw7g: secret "image-registry-tls" not found Jul 06 12:48:24.424375 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:24.424354 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls podName:d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:25.424333591 +0000 UTC m=+89.365174431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls") pod "image-registry-7fc7fc97d5-fgw7g" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5") : secret "image-registry-tls" not found Jul 06 12:48:24.935706 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:24.935666 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" event={"ID":"826fcfd6-3c84-4370-a599-5862b66e91f5","Type":"ContainerStarted","Data":"ddf6396686259261e04af870d5ad866cbf658f25c851aa7f6a853aa86de7b52b"} Jul 06 12:48:24.937165 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:24.937132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" event={"ID":"3ec35516-c4f2-40e3-bbae-fb91296ddf06","Type":"ContainerStarted","Data":"6c07cc410800758e6a2e99268fd84b3e26e6bfa2e06b54dc538bbfa46c0a49b7"} Jul 06 12:48:25.332449 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:25.332367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:25.332599 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:25.332555 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 12:48:25.332679 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:25.332655 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls podName:0fef59b9-4f17-45f7-8fee-a14b48278706 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:27.332613523 +0000 UTC m=+91.273454364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-fz2l4" (UID: "0fef59b9-4f17-45f7-8fee-a14b48278706") : secret "samples-operator-tls" not found Jul 06 12:48:25.432962 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:25.432925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:25.433133 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:25.433110 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:48:25.433201 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:25.433137 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc7fc97d5-fgw7g: secret "image-registry-tls" not found Jul 06 12:48:25.433250 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:25.433212 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls podName:d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:27.433191589 +0000 UTC m=+91.374032432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls") pod "image-registry-7fc7fc97d5-fgw7g" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5") : secret "image-registry-tls" not found Jul 06 12:48:26.944689 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:26.944644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" event={"ID":"3ec35516-c4f2-40e3-bbae-fb91296ddf06","Type":"ContainerStarted","Data":"c965d66fe83ec36943088d3e9a5a76e7fa0696d6c734eb44af5047496437ec89"} Jul 06 12:48:26.946198 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:26.946166 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" event={"ID":"826fcfd6-3c84-4370-a599-5862b66e91f5","Type":"ContainerStarted","Data":"9e2a03bbc4863641a6988176653e6f64d893a209954accb74748996ecb9a97f5"} Jul 06 12:48:26.972933 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:26.972886 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" podStartSLOduration=1.88104699 podStartE2EDuration="3.972873371s" totalImagesPullingTime="2.091826381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:26.972849759 +0000 UTC m=+90.913690618" watchObservedRunningTime="2026-07-06 12:48:26.972873371 +0000 UTC m=+90.913714230" Jul 06 12:48:26.997607 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:26.997568 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-l5mj5" podStartSLOduration=2.128433888 podStartE2EDuration="3.997553653s" totalImagesPullingTime="1.869119765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:26.997169232 +0000 UTC m=+90.938010094" watchObservedRunningTime="2026-07-06 12:48:26.997553653 +0000 UTC m=+90.938394526" Jul 06 12:48:27.346685 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.346577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:27.346828 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:27.346785 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 12:48:27.346867 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:27.346851 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls podName:0fef59b9-4f17-45f7-8fee-a14b48278706 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:31.346832587 +0000 UTC m=+95.287673434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-fz2l4" (UID: "0fef59b9-4f17-45f7-8fee-a14b48278706") : secret "samples-operator-tls" not found Jul 06 12:48:27.447252 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.447226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:27.447395 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:27.447333 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:48:27.447395 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:27.447344 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc7fc97d5-fgw7g: secret "image-registry-tls" not found Jul 06 12:48:27.447395 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:27.447389 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls podName:d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:31.447375601 +0000 UTC m=+95.388216437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls") pod "image-registry-7fc7fc97d5-fgw7g" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5") : secret "image-registry-tls" not found Jul 06 12:48:27.803334 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.803305 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx"] Jul 06 12:48:27.806461 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.806445 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" Jul 06 12:48:27.810821 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.810801 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 12:48:27.811511 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.811493 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xkrwh\"" Jul 06 12:48:27.811600 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.811579 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 12:48:27.827605 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.827585 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx"] Jul 06 12:48:27.950124 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:27.950093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpj2\" (UniqueName: \"kubernetes.io/projected/3772227c-7e8d-435a-8e31-9043a3903802-kube-api-access-qjpj2\") pod \"migrator-d57d58cc9-s4jrx\" (UID: \"3772227c-7e8d-435a-8e31-9043a3903802\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" Jul 06 12:48:28.050690 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.050651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpj2\" (UniqueName: \"kubernetes.io/projected/3772227c-7e8d-435a-8e31-9043a3903802-kube-api-access-qjpj2\") pod \"migrator-d57d58cc9-s4jrx\" (UID: \"3772227c-7e8d-435a-8e31-9043a3903802\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" Jul 06 12:48:28.288295 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.288263 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpj2\" (UniqueName: \"kubernetes.io/projected/3772227c-7e8d-435a-8e31-9043a3903802-kube-api-access-qjpj2\") pod \"migrator-d57d58cc9-s4jrx\" (UID: \"3772227c-7e8d-435a-8e31-9043a3903802\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" Jul 06 12:48:28.360074 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.360038 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lwksb"] Jul 06 12:48:28.363551 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.363529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.367988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.367971 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 12:48:28.368101 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.367977 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 12:48:28.368150 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.368122 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h754f\"" Jul 06 12:48:28.401489 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.401462 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lwksb"] Jul 06 12:48:28.416288 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.416268 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" Jul 06 12:48:28.454660 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.454617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/957f84d8-8eb9-471b-9ead-476eed6c7835-crio-socket\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.454861 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.454673 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.454861 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.454690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.454861 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.454707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxbpv\" (UniqueName: \"kubernetes.io/projected/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-api-access-jxbpv\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.454861 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.454741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/957f84d8-8eb9-471b-9ead-476eed6c7835-data-volume\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.555481 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.555418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/957f84d8-8eb9-471b-9ead-476eed6c7835-crio-socket\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.555481 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.555451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.555481 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.555471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.555727 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.555539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxbpv\" (UniqueName: \"kubernetes.io/projected/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-api-access-jxbpv\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.555727 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.555545 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/957f84d8-8eb9-471b-9ead-476eed6c7835-crio-socket\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.555727 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:28.555555 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:48:28.555727 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.555598 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/957f84d8-8eb9-471b-9ead-476eed6c7835-data-volume\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.555727 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:28.555556 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:48:28.555727 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:28.555679 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:29.055658762 +0000 UTC m=+92.996499628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : configmap "kube-rbac-proxy" not found Jul 06 12:48:28.555727 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:28.555726 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:29.055716306 +0000 UTC m=+92.996557144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : secret "insights-runtime-extractor-tls" not found Jul 06 12:48:28.555983 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.555892 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/957f84d8-8eb9-471b-9ead-476eed6c7835-data-volume\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.565448 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.565421 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx"] Jul 06 12:48:28.568443 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:28.568416 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3772227c_7e8d_435a_8e31_9043a3903802.slice/crio-6a9d6b01942d9d7867d52a20a79c297c09fc0263c3ff2d8f550e0a0cd8f90d52 WatchSource:0}: Error finding container 6a9d6b01942d9d7867d52a20a79c297c09fc0263c3ff2d8f550e0a0cd8f90d52: Status 404 returned error can't find the container with id 6a9d6b01942d9d7867d52a20a79c297c09fc0263c3ff2d8f550e0a0cd8f90d52 Jul 06 12:48:28.583205 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.583182 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxbpv\" (UniqueName: \"kubernetes.io/projected/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-api-access-jxbpv\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:28.956959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:28.956926 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" event={"ID":"3772227c-7e8d-435a-8e31-9043a3903802","Type":"ContainerStarted","Data":"6a9d6b01942d9d7867d52a20a79c297c09fc0263c3ff2d8f550e0a0cd8f90d52"} Jul 06 12:48:29.058772 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:29.058742 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:29.058772 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:29.058771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:29.058942 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:29.058860 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:48:29.058942 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:29.058874 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:48:29.058942 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:29.058908 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:30.05889496 +0000 UTC m=+93.999735798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : secret "insights-runtime-extractor-tls" not found Jul 06 12:48:29.058942 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:29.058940 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:30.058927071 +0000 UTC m=+93.999767908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : configmap "kube-rbac-proxy" not found Jul 06 12:48:29.960927 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:29.960902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" event={"ID":"3772227c-7e8d-435a-8e31-9043a3903802","Type":"ContainerStarted","Data":"8251720fc9e9a8f124e5f846a8d487e328c59dbe6658726f58628d64aa521784"} Jul 06 12:48:30.068301 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:30.068269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:30.068301 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:30.068304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:30.068460 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:30.068386 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:48:30.068460 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:30.068411 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:48:30.068460 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:30.068444 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:32.068429768 +0000 UTC m=+96.009270608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : configmap "kube-rbac-proxy" not found Jul 06 12:48:30.068460 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:30.068458 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:32.068451916 +0000 UTC m=+96.009292753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : secret "insights-runtime-extractor-tls" not found Jul 06 12:48:30.965713 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:30.965679 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" event={"ID":"3772227c-7e8d-435a-8e31-9043a3903802","Type":"ContainerStarted","Data":"ab83d3493e47bdd0817659975cd7d4f2792acab5ebb6e18d403dcc9e9959f5f2"} Jul 06 12:48:30.996193 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:30.996139 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-s4jrx" podStartSLOduration=2.7119042650000003 podStartE2EDuration="3.996122098s" totalImagesPullingTime="1.284217833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:30.994861746 +0000 UTC m=+94.935702604" watchObservedRunningTime="2026-07-06 12:48:30.996122098 +0000 UTC m=+94.936962964" Jul 06 12:48:31.342301 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:31.342233 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbb7x_d2168b17-6206-410a-a930-f665ddda267a/dns-node-resolver/0.log" Jul 06 12:48:31.379688 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:31.379661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:31.379819 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:31.379804 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 12:48:31.379876 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:31.379866 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls podName:0fef59b9-4f17-45f7-8fee-a14b48278706 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:39.379847285 +0000 UTC m=+103.320688122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-fz2l4" (UID: "0fef59b9-4f17-45f7-8fee-a14b48278706") : secret "samples-operator-tls" not found Jul 06 12:48:31.480484 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:31.480459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:31.480597 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:31.480555 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 12:48:31.480597 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:31.480565 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc7fc97d5-fgw7g: secret "image-registry-tls" not found Jul 06 12:48:31.480689 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:31.480605 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls podName:d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:39.480594458 +0000 UTC m=+103.421435296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls") pod "image-registry-7fc7fc97d5-fgw7g" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5") : secret "image-registry-tls" not found Jul 06 12:48:32.086296 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:32.086256 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:32.086296 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:32.086296 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:32.086767 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:32.086367 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:48:32.086767 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:32.086384 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:48:32.086767 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:32.086434 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:36.086418964 +0000 UTC m=+100.027259800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : configmap "kube-rbac-proxy" not found Jul 06 12:48:32.086767 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:32.086447 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:36.086441312 +0000 UTC m=+100.027282149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : secret "insights-runtime-extractor-tls" not found Jul 06 12:48:32.145690 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:32.145665 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9x5md_efdb6ed1-7693-4154-8cd4-94005582afff/node-ca/0.log" Jul 06 12:48:32.953265 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:32.953230 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8fp5_77930e36-25bd-46b1-9808-8c87f40fc07e/serve-healthcheck-canary/0.log" Jul 06 12:48:33.350570 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:33.350543 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-s4jrx_3772227c-7e8d-435a-8e31-9043a3903802/migrator/0.log" Jul 06 12:48:33.540300 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:33.540270 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-s4jrx_3772227c-7e8d-435a-8e31-9043a3903802/graceful-termination/0.log" Jul 06 12:48:35.311938 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:35.311865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:48:35.312291 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:35.311966 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 12:48:35.312291 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:35.312019 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls podName:3aba33d4-4f96-4774-8a16-cf7c2e4cead3 nodeName:}" failed. No retries permitted until 2026-07-06 12:49:39.312005924 +0000 UTC m=+163.252846760 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls") pod "dns-default-fvt4q" (UID: "3aba33d4-4f96-4774-8a16-cf7c2e4cead3") : secret "dns-default-metrics-tls" not found Jul 06 12:48:36.118343 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:36.118309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:36.118343 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:36.118342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:36.118569 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:36.118427 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 12:48:36.118569 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:36.118474 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 12:48:36.118569 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:36.118491 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:44.118476245 +0000 UTC m=+108.059317081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : configmap "kube-rbac-proxy" not found Jul 06 12:48:36.118569 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:48:36.118512 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls podName:957f84d8-8eb9-471b-9ead-476eed6c7835 nodeName:}" failed. No retries permitted until 2026-07-06 12:48:44.118501933 +0000 UTC m=+108.059342771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lwksb" (UID: "957f84d8-8eb9-471b-9ead-476eed6c7835") : secret "insights-runtime-extractor-tls" not found Jul 06 12:48:39.445173 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.445127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:39.447608 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.447589 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fef59b9-4f17-45f7-8fee-a14b48278706-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-fz2l4\" (UID: \"0fef59b9-4f17-45f7-8fee-a14b48278706\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:39.525168 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.525133 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" Jul 06 12:48:39.546002 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.545978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:39.548323 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.548298 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"image-registry-7fc7fc97d5-fgw7g\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:39.635121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.635088 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:39.667543 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.667514 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4"] Jul 06 12:48:39.775428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.775407 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc7fc97d5-fgw7g"] Jul 06 12:48:39.777351 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:39.777326 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e896c3_1a2a_48f1_8c42_7c4a8112b7a5.slice/crio-76bb5ca6423a6fba0bde53b1a14be22cabfd42b578c5c0d1f8f8a093d3af26d3 WatchSource:0}: Error finding container 76bb5ca6423a6fba0bde53b1a14be22cabfd42b578c5c0d1f8f8a093d3af26d3: Status 404 returned error can't find the container with id 76bb5ca6423a6fba0bde53b1a14be22cabfd42b578c5c0d1f8f8a093d3af26d3 Jul 06 12:48:39.901183 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.901151 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w6dgr" Jul 06 12:48:39.991978 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.991857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" event={"ID":"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5","Type":"ContainerStarted","Data":"950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471"} Jul 06 12:48:39.991978 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.991970 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" event={"ID":"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5","Type":"ContainerStarted","Data":"76bb5ca6423a6fba0bde53b1a14be22cabfd42b578c5c0d1f8f8a093d3af26d3"} Jul 06 12:48:39.992217 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.991998 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:48:39.993134 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:39.993111 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" event={"ID":"0fef59b9-4f17-45f7-8fee-a14b48278706","Type":"ContainerStarted","Data":"160f90188ccfd69bb19cc20b6b9d14ef0fac71aa4443be54e1aa264e443adaf9"} Jul 06 12:48:43.005197 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:43.005125 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" event={"ID":"0fef59b9-4f17-45f7-8fee-a14b48278706","Type":"ContainerStarted","Data":"57f07615f3eed4afd2b0db3e567733db73810a60af4060121eafa76bd4fbda6e"} Jul 06 12:48:43.005197 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:43.005175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" event={"ID":"0fef59b9-4f17-45f7-8fee-a14b48278706","Type":"ContainerStarted","Data":"549b4d77a6ef8dac1ac62c951b51d4b57061c64b1fdf2fe65ff753d67ae014a0"} Jul 06 12:48:43.033579 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:43.033534 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-fz2l4" podStartSLOduration=17.101024932 podStartE2EDuration="20.03351932s" totalImagesPullingTime="2.932494388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:43.031988428 +0000 UTC m=+106.972829267" watchObservedRunningTime="2026-07-06 12:48:43.03351932 +0000 UTC m=+106.974360178" Jul 06 12:48:43.033729 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:43.033703 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" podStartSLOduration=20.033698236 podStartE2EDuration="20.033698236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:40.027018685 +0000 UTC m=+103.967859545" watchObservedRunningTime="2026-07-06 12:48:43.033698236 +0000 UTC m=+106.974539095" Jul 06 12:48:44.179690 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:44.179655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:44.179690 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:44.179693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:44.180250 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:44.180229 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/957f84d8-8eb9-471b-9ead-476eed6c7835-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:44.182096 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:44.182074 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/957f84d8-8eb9-471b-9ead-476eed6c7835-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lwksb\" (UID: \"957f84d8-8eb9-471b-9ead-476eed6c7835\") " pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:44.273260 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:44.273231 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lwksb" Jul 06 12:48:44.406052 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:44.406025 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lwksb"] Jul 06 12:48:44.408736 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:44.408708 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957f84d8_8eb9_471b_9ead_476eed6c7835.slice/crio-274c7e8b12dfa08ae9c76311d617727a2c0e62cb1e9bcc90d31002ed900f6a38 WatchSource:0}: Error finding container 274c7e8b12dfa08ae9c76311d617727a2c0e62cb1e9bcc90d31002ed900f6a38: Status 404 returned error can't find the container with id 274c7e8b12dfa08ae9c76311d617727a2c0e62cb1e9bcc90d31002ed900f6a38 Jul 06 12:48:45.015545 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:45.015504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lwksb" event={"ID":"957f84d8-8eb9-471b-9ead-476eed6c7835","Type":"ContainerStarted","Data":"274c7e8b12dfa08ae9c76311d617727a2c0e62cb1e9bcc90d31002ed900f6a38"} Jul 06 12:48:49.032075 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:49.031996 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lwksb" event={"ID":"957f84d8-8eb9-471b-9ead-476eed6c7835","Type":"ContainerStarted","Data":"5863f59661b833fd350db850c5accc2e0dd13b19f94984bcd4692dffc3e21cae"} Jul 06 12:48:49.032075 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:49.032075 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lwksb" event={"ID":"957f84d8-8eb9-471b-9ead-476eed6c7835","Type":"ContainerStarted","Data":"98352a66b35fe520c3e1116517dff7861614b8b46d4edc3cf108ecba7ccd6c84"} Jul 06 12:48:52.044720 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:52.044684 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lwksb" event={"ID":"957f84d8-8eb9-471b-9ead-476eed6c7835","Type":"ContainerStarted","Data":"ed103d8a88e5402562179c71ab7d408e28745b8b56c3a2ff9e15de8c8348f146"} Jul 06 12:48:52.077128 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:52.077080 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lwksb" podStartSLOduration=17.447265835 podStartE2EDuration="24.077066064s" totalImagesPullingTime="6.629800229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:48:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:52.076389027 +0000 UTC m=+116.017229908" watchObservedRunningTime="2026-07-06 12:48:52.077066064 +0000 UTC m=+116.017906938" Jul 06 12:48:53.787929 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.787897 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl"] Jul 06 12:48:53.791048 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.791031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:53.797285 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.797259 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 12:48:53.797408 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.797269 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 12:48:53.798158 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.798138 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jzknd\"" Jul 06 12:48:53.798268 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.798182 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 12:48:53.803961 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.803944 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 12:48:53.863306 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.863279 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc7fc97d5-fgw7g"] Jul 06 12:48:53.864183 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.864158 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl"] Jul 06 12:48:53.960280 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.960245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da0c930f-c43d-4715-b77c-466af80b2b11-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54778c8d98-8zdbl\" (UID: \"da0c930f-c43d-4715-b77c-466af80b2b11\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:53.960435 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.960285 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjmr\" (UniqueName: \"kubernetes.io/projected/da0c930f-c43d-4715-b77c-466af80b2b11-kube-api-access-fhjmr\") pod \"managed-serviceaccount-addon-agent-54778c8d98-8zdbl\" (UID: \"da0c930f-c43d-4715-b77c-466af80b2b11\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:53.976966 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.976912 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75d59b4dc6-b65v7"] Jul 06 12:48:53.980704 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:53.980683 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.025579 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.025556 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75d59b4dc6-b65v7"] Jul 06 12:48:54.060830 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.060780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da0c930f-c43d-4715-b77c-466af80b2b11-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54778c8d98-8zdbl\" (UID: \"da0c930f-c43d-4715-b77c-466af80b2b11\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:54.060830 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.060807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjmr\" (UniqueName: \"kubernetes.io/projected/da0c930f-c43d-4715-b77c-466af80b2b11-kube-api-access-fhjmr\") pod \"managed-serviceaccount-addon-agent-54778c8d98-8zdbl\" (UID: \"da0c930f-c43d-4715-b77c-466af80b2b11\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:54.063285 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.063260 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da0c930f-c43d-4715-b77c-466af80b2b11-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54778c8d98-8zdbl\" (UID: \"da0c930f-c43d-4715-b77c-466af80b2b11\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:54.087750 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.087727 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjmr\" (UniqueName: \"kubernetes.io/projected/da0c930f-c43d-4715-b77c-466af80b2b11-kube-api-access-fhjmr\") pod \"managed-serviceaccount-addon-agent-54778c8d98-8zdbl\" (UID: \"da0c930f-c43d-4715-b77c-466af80b2b11\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:54.107518 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.107500 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.162991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvrtx\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-kube-api-access-cvrtx\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.163034 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-registry-tls\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.163078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12c955fc-4b31-41c1-999d-b5191af3b226-image-registry-private-configuration\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.163104 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12c955fc-4b31-41c1-999d-b5191af3b226-ca-trust-extracted\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.163129 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12c955fc-4b31-41c1-999d-b5191af3b226-installation-pull-secrets\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.163160 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12c955fc-4b31-41c1-999d-b5191af3b226-trusted-ca\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.163225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12c955fc-4b31-41c1-999d-b5191af3b226-registry-certificates\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.164085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.163246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-bound-sa-token\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.263933 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.263901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12c955fc-4b31-41c1-999d-b5191af3b226-image-registry-private-configuration\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.263933 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.263935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12c955fc-4b31-41c1-999d-b5191af3b226-ca-trust-extracted\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.264148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.263953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12c955fc-4b31-41c1-999d-b5191af3b226-installation-pull-secrets\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.264148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.263980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12c955fc-4b31-41c1-999d-b5191af3b226-trusted-ca\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.264148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.264046 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12c955fc-4b31-41c1-999d-b5191af3b226-registry-certificates\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.264148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.264070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-bound-sa-token\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.264148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.264105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvrtx\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-kube-api-access-cvrtx\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.264148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.264135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-registry-tls\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.264440 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.264353 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12c955fc-4b31-41c1-999d-b5191af3b226-ca-trust-extracted\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.265141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.265115 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12c955fc-4b31-41c1-999d-b5191af3b226-trusted-ca\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.265141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.265124 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12c955fc-4b31-41c1-999d-b5191af3b226-registry-certificates\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.266496 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.266474 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12c955fc-4b31-41c1-999d-b5191af3b226-installation-pull-secrets\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.267029 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.267007 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12c955fc-4b31-41c1-999d-b5191af3b226-image-registry-private-configuration\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.267096 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.267082 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-registry-tls\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.275103 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.275083 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl"] Jul 06 12:48:54.277816 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:54.277794 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda0c930f_c43d_4715_b77c_466af80b2b11.slice/crio-2b8f803eed907d80c99d647189ece24fa5ec3c525c635b6da071c2f596b49799 WatchSource:0}: Error finding container 2b8f803eed907d80c99d647189ece24fa5ec3c525c635b6da071c2f596b49799: Status 404 returned error can't find the container with id 2b8f803eed907d80c99d647189ece24fa5ec3c525c635b6da071c2f596b49799 Jul 06 12:48:54.279796 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.279773 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-bound-sa-token\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.282575 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.282552 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvrtx\" (UniqueName: \"kubernetes.io/projected/12c955fc-4b31-41c1-999d-b5191af3b226-kube-api-access-cvrtx\") pod \"image-registry-75d59b4dc6-b65v7\" (UID: \"12c955fc-4b31-41c1-999d-b5191af3b226\") " pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.288878 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.288858 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:54.432002 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:54.431972 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75d59b4dc6-b65v7"] Jul 06 12:48:54.434802 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:48:54.434777 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c955fc_4b31_41c1_999d_b5191af3b226.slice/crio-b7bd459b71cdf205629a122dcc178e2f43cda7a50be4f6d15d041f7863c00887 WatchSource:0}: Error finding container b7bd459b71cdf205629a122dcc178e2f43cda7a50be4f6d15d041f7863c00887: Status 404 returned error can't find the container with id b7bd459b71cdf205629a122dcc178e2f43cda7a50be4f6d15d041f7863c00887 Jul 06 12:48:55.056409 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:55.056369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" event={"ID":"12c955fc-4b31-41c1-999d-b5191af3b226","Type":"ContainerStarted","Data":"c7381aa133c49c3b715ccd783cb31c68611ed0d46d3870e9b53ceb9921ca84fa"} Jul 06 12:48:55.057034 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:55.056431 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" event={"ID":"12c955fc-4b31-41c1-999d-b5191af3b226","Type":"ContainerStarted","Data":"b7bd459b71cdf205629a122dcc178e2f43cda7a50be4f6d15d041f7863c00887"} Jul 06 12:48:55.057034 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:55.056506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:48:55.058008 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:55.057976 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" event={"ID":"da0c930f-c43d-4715-b77c-466af80b2b11","Type":"ContainerStarted","Data":"2b8f803eed907d80c99d647189ece24fa5ec3c525c635b6da071c2f596b49799"} Jul 06 12:48:57.066694 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:57.066601 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" event={"ID":"da0c930f-c43d-4715-b77c-466af80b2b11","Type":"ContainerStarted","Data":"044941d15859e2785c8597bd2688ec81595712fd77ef9f648b74114ac5491df1"} Jul 06 12:48:57.089563 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:57.089518 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54778c8d98-8zdbl" podStartSLOduration=1.7491764810000001 podStartE2EDuration="4.089505843s" totalImagesPullingTime="2.340329362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:57.088618915 +0000 UTC m=+121.029459770" watchObservedRunningTime="2026-07-06 12:48:57.089505843 +0000 UTC m=+121.030346702" Jul 06 12:48:57.090335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:57.090309 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" podStartSLOduration=4.090298963 podStartE2EDuration="4.090298963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:48:55.236072863 +0000 UTC m=+119.176913748" watchObservedRunningTime="2026-07-06 12:48:57.090298963 +0000 UTC m=+121.031139822" Jul 06 12:48:59.562241 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.562210 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf"] Jul 06 12:48:59.565289 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.565273 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.570161 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.570140 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 12:48:59.570275 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.570251 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 12:48:59.570349 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.570286 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 12:48:59.570349 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.570326 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kbqrn\"" Jul 06 12:48:59.570448 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.570360 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 12:48:59.570598 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.570584 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 12:48:59.595227 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.595201 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf"] Jul 06 12:48:59.603596 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.603575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16d7bc32-c1ca-46d1-a368-8efd579219a4-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.603722 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.603617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg86x\" (UniqueName: \"kubernetes.io/projected/16d7bc32-c1ca-46d1-a368-8efd579219a4-kube-api-access-tg86x\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.603722 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.603678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16d7bc32-c1ca-46d1-a368-8efd579219a4-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.603830 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.603747 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16d7bc32-c1ca-46d1-a368-8efd579219a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.704728 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.704703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg86x\" (UniqueName: \"kubernetes.io/projected/16d7bc32-c1ca-46d1-a368-8efd579219a4-kube-api-access-tg86x\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.704831 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.704745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16d7bc32-c1ca-46d1-a368-8efd579219a4-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.704897 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.704878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16d7bc32-c1ca-46d1-a368-8efd579219a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.704983 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.704931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16d7bc32-c1ca-46d1-a368-8efd579219a4-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.705349 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.705332 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16d7bc32-c1ca-46d1-a368-8efd579219a4-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.707345 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.707324 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16d7bc32-c1ca-46d1-a368-8efd579219a4-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.707434 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.707343 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16d7bc32-c1ca-46d1-a368-8efd579219a4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.715977 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.715958 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg86x\" (UniqueName: \"kubernetes.io/projected/16d7bc32-c1ca-46d1-a368-8efd579219a4-kube-api-access-tg86x\") pod \"prometheus-operator-567f7b8b7c-vf4zf\" (UID: \"16d7bc32-c1ca-46d1-a368-8efd579219a4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:48:59.874369 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:48:59.874348 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" Jul 06 12:49:00.004764 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:00.004736 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf"] Jul 06 12:49:00.007451 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:49:00.007418 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d7bc32_c1ca_46d1_a368_8efd579219a4.slice/crio-c855b3307a92b60becd1bfafb87090a934b71a97f74422b45fc024f166701462 WatchSource:0}: Error finding container c855b3307a92b60becd1bfafb87090a934b71a97f74422b45fc024f166701462: Status 404 returned error can't find the container with id c855b3307a92b60becd1bfafb87090a934b71a97f74422b45fc024f166701462 Jul 06 12:49:00.081532 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:00.081498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" event={"ID":"16d7bc32-c1ca-46d1-a368-8efd579219a4","Type":"ContainerStarted","Data":"c855b3307a92b60becd1bfafb87090a934b71a97f74422b45fc024f166701462"} Jul 06 12:49:02.089444 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:02.089406 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" event={"ID":"16d7bc32-c1ca-46d1-a368-8efd579219a4","Type":"ContainerStarted","Data":"09451e308f616cbfc6c417c274e30a4df05c3bd2e029c78280b30774bf78bfab"} Jul 06 12:49:02.089866 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:02.089461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" event={"ID":"16d7bc32-c1ca-46d1-a368-8efd579219a4","Type":"ContainerStarted","Data":"93a866003d63911a372727054850e114bcf63a1ee9ef4cc21b52a846931b9cb9"} Jul 06 12:49:02.110247 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:02.110200 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vf4zf" podStartSLOduration=1.689481605 podStartE2EDuration="3.110187322s" totalImagesPullingTime="1.420705717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:49:02.108915149 +0000 UTC m=+126.049756021" watchObservedRunningTime="2026-07-06 12:49:02.110187322 +0000 UTC m=+126.051028199" Jul 06 12:49:03.869529 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:03.869503 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:49:04.050929 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.050894 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x"] Jul 06 12:49:04.054139 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.054124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.057499 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.057478 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 12:49:04.057850 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.057834 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-78qb6\"" Jul 06 12:49:04.058863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.058846 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 12:49:04.059186 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.059171 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 12:49:04.080396 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.080365 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x"] Jul 06 12:49:04.097445 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.097421 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fvj5b"] Jul 06 12:49:04.100561 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.100545 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.103431 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.103414 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dp8jt\"" Jul 06 12:49:04.103730 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.103709 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 12:49:04.103730 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.103727 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 12:49:04.103852 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.103715 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 12:49:04.137131 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137069 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/721ec354-d37f-4094-8bb8-f31468932417-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.137131 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137104 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/721ec354-d37f-4094-8bb8-f31468932417-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.137131 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137123 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.137350 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-textfile\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137350 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137217 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137350 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-sys\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137350 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137291 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-tls\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137350 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137324 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.137510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137356 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-wtmp\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137373 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmhb\" (UniqueName: \"kubernetes.io/projected/b1c83a55-eedf-424d-b7f1-051e78ae51ba-kube-api-access-khmhb\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137409 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htlb\" (UniqueName: \"kubernetes.io/projected/721ec354-d37f-4094-8bb8-f31468932417-kube-api-access-9htlb\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.137510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.137625 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137505 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c83a55-eedf-424d-b7f1-051e78ae51ba-metrics-client-ca\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137625 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137534 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.137625 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.137554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-root\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238221 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238179 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c83a55-eedf-424d-b7f1-051e78ae51ba-metrics-client-ca\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238221 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-root\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/721ec354-d37f-4094-8bb8-f31468932417-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.238463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/721ec354-d37f-4094-8bb8-f31468932417-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.238463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238447 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-root\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238700 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.238700 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-textfile\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238700 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238560 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238700 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-sys\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238700 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-tls\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238700 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.238985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-wtmp\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmhb\" (UniqueName: \"kubernetes.io/projected/b1c83a55-eedf-424d-b7f1-051e78ae51ba-kube-api-access-khmhb\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htlb\" (UniqueName: \"kubernetes.io/projected/721ec354-d37f-4094-8bb8-f31468932417-kube-api-access-9htlb\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.238985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.238985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238849 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-textfile\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238869 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c83a55-eedf-424d-b7f1-051e78ae51ba-metrics-client-ca\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.238985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.238944 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-accelerators-collector-config\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.239230 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.239154 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.239230 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.239208 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/721ec354-d37f-4094-8bb8-f31468932417-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.239361 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.239344 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-wtmp\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.239421 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.239366 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1c83a55-eedf-424d-b7f1-051e78ae51ba-sys\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.239548 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.239526 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/721ec354-d37f-4094-8bb8-f31468932417-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.239718 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:49:04.239701 2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 06 12:49:04.239812 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:49:04.239792 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-tls podName:721ec354-d37f-4094-8bb8-f31468932417 nodeName:}" failed. No retries permitted until 2026-07-06 12:49:04.739770878 +0000 UTC m=+128.680611728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-gfz7x" (UID: "721ec354-d37f-4094-8bb8-f31468932417") : secret "kube-state-metrics-tls" not found Jul 06 12:49:04.241325 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.241292 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.241432 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.241415 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.241602 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.241578 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1c83a55-eedf-424d-b7f1-051e78ae51ba-node-exporter-tls\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.256194 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.256166 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmhb\" (UniqueName: \"kubernetes.io/projected/b1c83a55-eedf-424d-b7f1-051e78ae51ba-kube-api-access-khmhb\") pod \"node-exporter-fvj5b\" (UID: \"b1c83a55-eedf-424d-b7f1-051e78ae51ba\") " pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.256290 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.256237 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htlb\" (UniqueName: \"kubernetes.io/projected/721ec354-d37f-4094-8bb8-f31468932417-kube-api-access-9htlb\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.408964 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.408880 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fvj5b" Jul 06 12:49:04.420794 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:49:04.420767 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c83a55_eedf_424d_b7f1_051e78ae51ba.slice/crio-11f122b8130b49492e7d18d5338ff38df2fcb3aea0b66e7af2b2b23866ca5168 WatchSource:0}: Error finding container 11f122b8130b49492e7d18d5338ff38df2fcb3aea0b66e7af2b2b23866ca5168: Status 404 returned error can't find the container with id 11f122b8130b49492e7d18d5338ff38df2fcb3aea0b66e7af2b2b23866ca5168 Jul 06 12:49:04.742746 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.742654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.745108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.745089 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/721ec354-d37f-4094-8bb8-f31468932417-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfz7x\" (UID: \"721ec354-d37f-4094-8bb8-f31468932417\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:04.962771 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:04.962735 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" Jul 06 12:49:05.102294 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:05.102263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fvj5b" event={"ID":"b1c83a55-eedf-424d-b7f1-051e78ae51ba","Type":"ContainerStarted","Data":"11f122b8130b49492e7d18d5338ff38df2fcb3aea0b66e7af2b2b23866ca5168"} Jul 06 12:49:05.119444 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:05.119416 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x"] Jul 06 12:49:05.122611 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:49:05.122571 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721ec354_d37f_4094_8bb8_f31468932417.slice/crio-eeff5eead2cf69096269f3f9b9ce33e9c479c5075908e1d5b3f0cd9b1b0a2f7d WatchSource:0}: Error finding container eeff5eead2cf69096269f3f9b9ce33e9c479c5075908e1d5b3f0cd9b1b0a2f7d: Status 404 returned error can't find the container with id eeff5eead2cf69096269f3f9b9ce33e9c479c5075908e1d5b3f0cd9b1b0a2f7d Jul 06 12:49:06.107060 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.107024 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" event={"ID":"721ec354-d37f-4094-8bb8-f31468932417","Type":"ContainerStarted","Data":"eeff5eead2cf69096269f3f9b9ce33e9c479c5075908e1d5b3f0cd9b1b0a2f7d"} Jul 06 12:49:06.108814 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.108778 2568 generic.go:356] "Generic (PLEG): container finished" podID="b1c83a55-eedf-424d-b7f1-051e78ae51ba" containerID="4d916cd46071d3c1e9f0e0a66eb1f8049230f211a7418dcdffa53367abbf6810" exitCode=0 Jul 06 12:49:06.108938 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.108822 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fvj5b" event={"ID":"b1c83a55-eedf-424d-b7f1-051e78ae51ba","Type":"ContainerDied","Data":"4d916cd46071d3c1e9f0e0a66eb1f8049230f211a7418dcdffa53367abbf6810"} Jul 06 12:49:06.357719 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.357609 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:49:06.360349 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.360320 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d42744ee-0140-41b4-afba-64fd0483ee84-metrics-certs\") pod \"network-metrics-daemon-8n477\" (UID: \"d42744ee-0140-41b4-afba-64fd0483ee84\") " pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:49:06.504241 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.504221 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dxdc8\"" Jul 06 12:49:06.512503 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.512481 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8n477" Jul 06 12:49:06.684869 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:06.684837 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8n477"] Jul 06 12:49:06.695468 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:49:06.695363 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42744ee_0140_41b4_afba_64fd0483ee84.slice/crio-4c60bf9f25a0360071631fb87db09ab1cbdca4411d53814a6ee0f76f3f6b4017 WatchSource:0}: Error finding container 4c60bf9f25a0360071631fb87db09ab1cbdca4411d53814a6ee0f76f3f6b4017: Status 404 returned error can't find the container with id 4c60bf9f25a0360071631fb87db09ab1cbdca4411d53814a6ee0f76f3f6b4017 Jul 06 12:49:07.114923 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.114881 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" event={"ID":"721ec354-d37f-4094-8bb8-f31468932417","Type":"ContainerStarted","Data":"e1c785a5cd0779f2dc28af368bf9b7fc2af4b90090073738d654b2ce2544fea8"} Jul 06 12:49:07.115399 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.114975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" event={"ID":"721ec354-d37f-4094-8bb8-f31468932417","Type":"ContainerStarted","Data":"ee4a5036167a366b5630178e1adda2c57d92f08ec1d4fe37d9890738d2b0ea1b"} Jul 06 12:49:07.115399 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.115005 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" event={"ID":"721ec354-d37f-4094-8bb8-f31468932417","Type":"ContainerStarted","Data":"4a243c9695b59694898c3d4607a32d711c4b711793447b7b4b86db34167b495d"} Jul 06 12:49:07.116136 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.116113 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8n477" event={"ID":"d42744ee-0140-41b4-afba-64fd0483ee84","Type":"ContainerStarted","Data":"4c60bf9f25a0360071631fb87db09ab1cbdca4411d53814a6ee0f76f3f6b4017"} Jul 06 12:49:07.118155 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.118124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fvj5b" event={"ID":"b1c83a55-eedf-424d-b7f1-051e78ae51ba","Type":"ContainerStarted","Data":"2b30884af46103ebd489389dd1402d439be9893ce060b7b4dacbf61542c3e805"} Jul 06 12:49:07.118257 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.118166 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fvj5b" event={"ID":"b1c83a55-eedf-424d-b7f1-051e78ae51ba","Type":"ContainerStarted","Data":"4cc8cb569356a94fc9944bd0e957bc0944745f82d4632d885e81a8bc5ebbe2ce"} Jul 06 12:49:07.233615 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.233547 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfz7x" podStartSLOduration=1.853799228 podStartE2EDuration="3.233524628s" totalImagesPullingTime="1.3797254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:49:07.206927264 +0000 UTC m=+131.147768122" watchObservedRunningTime="2026-07-06 12:49:07.233524628 +0000 UTC m=+131.174365488" Jul 06 12:49:07.342393 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:07.342332 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fvj5b" podStartSLOduration=2.153318007 podStartE2EDuration="3.342311786s" totalImagesPullingTime="1.118866812s" totalInitContainerRuntime="70.126967ms" isStatefulPod=true podCreationTimestamp="2026-07-06 12:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:49:07.341611657 +0000 UTC m=+131.282452517" watchObservedRunningTime="2026-07-06 12:49:07.342311786 +0000 UTC m=+131.283152645" Jul 06 12:49:08.123168 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.123131 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8n477" event={"ID":"d42744ee-0140-41b4-afba-64fd0483ee84","Type":"ContainerStarted","Data":"e99bae1ef33820b371bdfd7d56275f0472a08f947a47c5f2a6db0c42fdb486f9"} Jul 06 12:49:08.123564 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.123186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8n477" event={"ID":"d42744ee-0140-41b4-afba-64fd0483ee84","Type":"ContainerStarted","Data":"d9b64f8a8e4fe24ae181954a906fee17d76100697752a6dd721bf82b2085c7bb"} Jul 06 12:49:08.153965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.153919 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8n477" podStartSLOduration=131.094200582 podStartE2EDuration="2m12.153904979s" totalImagesPullingTime="1.059704397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:49:08.153672969 +0000 UTC m=+132.094513830" watchObservedRunningTime="2026-07-06 12:49:08.153904979 +0000 UTC m=+132.094745839" Jul 06 12:49:08.810484 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.810446 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw"] Jul 06 12:49:08.814055 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.814033 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:08.816881 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.816858 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 12:49:08.818080 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.818060 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ckgcj\"" Jul 06 12:49:08.828352 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.828328 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw"] Jul 06 12:49:08.879129 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.879094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b310581-9dde-4970-9268-634eded04b9f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6sqlw\" (UID: \"3b310581-9dde-4970-9268-634eded04b9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:08.879257 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.879137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9544\" (UniqueName: \"kubernetes.io/projected/3b310581-9dde-4970-9268-634eded04b9f-kube-api-access-l9544\") pod \"monitoring-plugin-d5dcff7cf-6sqlw\" (UID: \"3b310581-9dde-4970-9268-634eded04b9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:08.980473 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.980442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b310581-9dde-4970-9268-634eded04b9f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6sqlw\" (UID: \"3b310581-9dde-4970-9268-634eded04b9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:08.980473 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.980475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9544\" (UniqueName: \"kubernetes.io/projected/3b310581-9dde-4970-9268-634eded04b9f-kube-api-access-l9544\") pod \"monitoring-plugin-d5dcff7cf-6sqlw\" (UID: \"3b310581-9dde-4970-9268-634eded04b9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:08.983108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.983075 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b310581-9dde-4970-9268-634eded04b9f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6sqlw\" (UID: \"3b310581-9dde-4970-9268-634eded04b9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:08.992087 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:08.992064 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9544\" (UniqueName: \"kubernetes.io/projected/3b310581-9dde-4970-9268-634eded04b9f-kube-api-access-l9544\") pod \"monitoring-plugin-d5dcff7cf-6sqlw\" (UID: \"3b310581-9dde-4970-9268-634eded04b9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:09.123571 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:09.123533 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:09.267868 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:09.267846 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw"] Jul 06 12:49:09.269293 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:49:09.269270 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b310581_9dde_4970_9268_634eded04b9f.slice/crio-ffa12343a7d62f22ca9f078914054e633dd14359ded4b62d676043a8d5bc4f0f WatchSource:0}: Error finding container ffa12343a7d62f22ca9f078914054e633dd14359ded4b62d676043a8d5bc4f0f: Status 404 returned error can't find the container with id ffa12343a7d62f22ca9f078914054e633dd14359ded4b62d676043a8d5bc4f0f Jul 06 12:49:10.132988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:10.132938 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" event={"ID":"3b310581-9dde-4970-9268-634eded04b9f","Type":"ContainerStarted","Data":"ffa12343a7d62f22ca9f078914054e633dd14359ded4b62d676043a8d5bc4f0f"} Jul 06 12:49:11.140189 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:11.139508 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" event={"ID":"3b310581-9dde-4970-9268-634eded04b9f","Type":"ContainerStarted","Data":"5ee3e0ce43f113483dabfc7e0e5a05ba03f05319dab77fdfdc69dfeedfd7fcd1"} Jul 06 12:49:11.140608 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:11.140290 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:11.145292 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:11.145264 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" Jul 06 12:49:11.364127 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:11.364069 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6sqlw" podStartSLOduration=1.869275909 podStartE2EDuration="3.364052244s" totalImagesPullingTime="1.494776335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:49:11.191022961 +0000 UTC m=+135.131863820" watchObservedRunningTime="2026-07-06 12:49:11.364052244 +0000 UTC m=+135.304893102" Jul 06 12:49:16.066722 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:16.066694 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75d59b4dc6-b65v7" Jul 06 12:49:18.883554 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:18.883504 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" podUID="d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" containerName="registry" containerID="cri-o://950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471" gracePeriod=30 Jul 06 12:49:19.115588 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.115567 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:49:19.173352 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173285 2568 generic.go:356] "Generic (PLEG): container finished" podID="d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" containerID="950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471" exitCode=0 Jul 06 12:49:19.173352 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173332 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" Jul 06 12:49:19.173529 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" event={"ID":"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5","Type":"ContainerDied","Data":"950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471"} Jul 06 12:49:19.173529 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173417 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc7fc97d5-fgw7g" event={"ID":"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5","Type":"ContainerDied","Data":"76bb5ca6423a6fba0bde53b1a14be22cabfd42b578c5c0d1f8f8a093d3af26d3"} Jul 06 12:49:19.173529 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173458 2568 scope.go:117] "RemoveContainer" containerID="950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471" Jul 06 12:49:19.173529 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173500 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-ca-trust-extracted\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.173744 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173530 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-certificates\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.173744 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173577 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-image-registry-private-configuration\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.173744 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173681 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.173744 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173731 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-bound-sa-token\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.173931 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173770 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-trusted-ca\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.173931 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173808 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-installation-pull-secrets\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.173931 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173868 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5pbh\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-kube-api-access-b5pbh\") pod \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\" (UID: \"d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5\") " Jul 06 12:49:19.174087 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.173940 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:49:19.174154 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.174136 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-certificates\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.174746 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.174717 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:49:19.176908 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.176880 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:49:19.177338 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.177313 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:49:19.177338 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.177315 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:49:19.177863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.177844 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-kube-api-access-b5pbh" (OuterVolumeSpecName: "kube-api-access-b5pbh") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "kube-api-access-b5pbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:49:19.178006 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.177925 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:49:19.182543 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.182524 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" (UID: "d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:49:19.193479 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.193459 2568 scope.go:117] "RemoveContainer" containerID="950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471" Jul 06 12:49:19.193801 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:49:19.193779 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471\": container with ID starting with 950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471 not found: ID does not exist" containerID="950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471" Jul 06 12:49:19.193892 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.193809 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471"} err="failed to get container status \"950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471\": rpc error: code = NotFound desc = could not find container \"950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471\": container with ID starting with 950738d0ebde5885f2c5553735471b708a05ef7a2368ef94cc64cc359a9a0471 not found: ID does not exist" Jul 06 12:49:19.275470 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.275447 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5pbh\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-kube-api-access-b5pbh\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.275470 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.275468 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-ca-trust-extracted\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.275592 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.275477 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-image-registry-private-configuration\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.275592 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.275487 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-registry-tls\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.275592 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.275495 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-bound-sa-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.275592 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.275502 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-trusted-ca\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.275592 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.275510 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5-installation-pull-secrets\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:49:19.550410 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.550381 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc7fc97d5-fgw7g"] Jul 06 12:49:19.556943 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:19.556920 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fc7fc97d5-fgw7g"] Jul 06 12:49:20.589877 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:20.589841 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" path="/var/lib/kubelet/pods/d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5/volumes" Jul 06 12:49:34.423579 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:49:34.423543 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fvt4q" podUID="3aba33d4-4f96-4774-8a16-cf7c2e4cead3" Jul 06 12:49:35.234735 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:35.234660 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvt4q" Jul 06 12:49:39.323106 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:39.323064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:49:39.325401 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:39.325372 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3aba33d4-4f96-4774-8a16-cf7c2e4cead3-metrics-tls\") pod \"dns-default-fvt4q\" (UID: \"3aba33d4-4f96-4774-8a16-cf7c2e4cead3\") " pod="openshift-dns/dns-default-fvt4q" Jul 06 12:49:39.438603 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:39.438580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9njlh\"" Jul 06 12:49:39.445918 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:39.445902 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvt4q" Jul 06 12:49:39.600309 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:39.600278 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fvt4q"] Jul 06 12:49:39.604650 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:49:39.604599 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aba33d4_4f96_4774_8a16_cf7c2e4cead3.slice/crio-fe78247cfeb7df8a247059d88d489a855b47e7baab172791c76e452990c261fe WatchSource:0}: Error finding container fe78247cfeb7df8a247059d88d489a855b47e7baab172791c76e452990c261fe: Status 404 returned error can't find the container with id fe78247cfeb7df8a247059d88d489a855b47e7baab172791c76e452990c261fe Jul 06 12:49:40.255162 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:40.255124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvt4q" event={"ID":"3aba33d4-4f96-4774-8a16-cf7c2e4cead3","Type":"ContainerStarted","Data":"fe78247cfeb7df8a247059d88d489a855b47e7baab172791c76e452990c261fe"} Jul 06 12:49:42.265028 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:42.264996 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvt4q" event={"ID":"3aba33d4-4f96-4774-8a16-cf7c2e4cead3","Type":"ContainerStarted","Data":"eafbc50f377259ea1388e62992e9384c8bc097463c3c4b2807c3e5562cae297f"} Jul 06 12:49:42.265404 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:42.265045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvt4q" event={"ID":"3aba33d4-4f96-4774-8a16-cf7c2e4cead3","Type":"ContainerStarted","Data":"dbc791effa841f646a4f2815ee143e6b5ad59cdcf6cdc85f9705734fb3048089"} Jul 06 12:49:42.265404 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:42.265141 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fvt4q" Jul 06 12:49:42.283877 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:42.283835 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fvt4q" podStartSLOduration=129.650454647 podStartE2EDuration="2m11.283820452s" totalImagesPullingTime="1.633365805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:49:42.282652369 +0000 UTC m=+166.223493220" watchObservedRunningTime="2026-07-06 12:49:42.283820452 +0000 UTC m=+166.224661316" Jul 06 12:49:48.290925 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:48.290891 2568 generic.go:356] "Generic (PLEG): container finished" podID="3ec35516-c4f2-40e3-bbae-fb91296ddf06" containerID="c965d66fe83ec36943088d3e9a5a76e7fa0696d6c734eb44af5047496437ec89" exitCode=0 Jul 06 12:49:48.291299 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:48.290956 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" event={"ID":"3ec35516-c4f2-40e3-bbae-fb91296ddf06","Type":"ContainerDied","Data":"c965d66fe83ec36943088d3e9a5a76e7fa0696d6c734eb44af5047496437ec89"} Jul 06 12:49:48.291299 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:48.291258 2568 scope.go:117] "RemoveContainer" containerID="c965d66fe83ec36943088d3e9a5a76e7fa0696d6c734eb44af5047496437ec89" Jul 06 12:49:49.296219 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:49.296183 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xnzpc" event={"ID":"3ec35516-c4f2-40e3-bbae-fb91296ddf06","Type":"ContainerStarted","Data":"f176b10f973f42e812315f5542d9ef493a3ae347b6ef113faa4451ba15eedb04"} Jul 06 12:49:52.271436 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:49:52.271407 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fvt4q" Jul 06 12:51:56.490616 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:51:56.490591 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:51:56.494311 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:51:56.494291 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:51:56.497075 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:51:56.497056 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 12:54:20.407408 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.407333 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx"] Jul 06 12:54:20.407889 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.407617 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" containerName="registry" Jul 06 12:54:20.407889 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.407642 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" containerName="registry" Jul 06 12:54:20.407889 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.407706 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e896c3-1a2a-48f1-8c42-7c4a8112b7a5" containerName="registry" Jul 06 12:54:20.410614 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.410599 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.415145 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.415125 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 12:54:20.416014 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.415993 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2pj8k\"" Jul 06 12:54:20.416132 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.416013 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 12:54:20.435500 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.435469 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx"] Jul 06 12:54:20.480072 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.480044 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.480198 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.480093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkzf\" (UniqueName: \"kubernetes.io/projected/2e41154f-5ed5-484b-99c2-b8b5c6c33511-kube-api-access-dnkzf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.480198 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.480177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.581400 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.581368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkzf\" (UniqueName: \"kubernetes.io/projected/2e41154f-5ed5-484b-99c2-b8b5c6c33511-kube-api-access-dnkzf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.581547 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.581423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.581547 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.581466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.581833 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.581817 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.581875 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.581858 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.590547 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.590521 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkzf\" (UniqueName: \"kubernetes.io/projected/2e41154f-5ed5-484b-99c2-b8b5c6c33511-kube-api-access-dnkzf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.719784 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.719695 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:20.847701 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.847654 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx"] Jul 06 12:54:20.851604 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:54:20.851573 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e41154f_5ed5_484b_99c2_b8b5c6c33511.slice/crio-e2ac7f7eb12f9df286ac05c0a0a0d939e32accd0c46174e94e3c3af9ad25d595 WatchSource:0}: Error finding container e2ac7f7eb12f9df286ac05c0a0a0d939e32accd0c46174e94e3c3af9ad25d595: Status 404 returned error can't find the container with id e2ac7f7eb12f9df286ac05c0a0a0d939e32accd0c46174e94e3c3af9ad25d595 Jul 06 12:54:20.853546 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:20.853532 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 12:54:21.238839 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:21.238802 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" event={"ID":"2e41154f-5ed5-484b-99c2-b8b5c6c33511","Type":"ContainerStarted","Data":"e2ac7f7eb12f9df286ac05c0a0a0d939e32accd0c46174e94e3c3af9ad25d595"} Jul 06 12:54:27.264171 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:27.264140 2568 generic.go:356] "Generic (PLEG): container finished" podID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerID="5551aebecfda9412645be86cc6b12a9c253b130517dbf9942f2f024f4f2cc8c6" exitCode=0 Jul 06 12:54:27.264479 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:27.264219 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" event={"ID":"2e41154f-5ed5-484b-99c2-b8b5c6c33511","Type":"ContainerDied","Data":"5551aebecfda9412645be86cc6b12a9c253b130517dbf9942f2f024f4f2cc8c6"} Jul 06 12:54:30.278397 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:30.278359 2568 generic.go:356] "Generic (PLEG): container finished" podID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerID="d46e3b638249c37770402e9c9632465109bc0d69f1525c7d92547768b7e4f42d" exitCode=0 Jul 06 12:54:30.278799 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:30.278419 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" event={"ID":"2e41154f-5ed5-484b-99c2-b8b5c6c33511","Type":"ContainerDied","Data":"d46e3b638249c37770402e9c9632465109bc0d69f1525c7d92547768b7e4f42d"} Jul 06 12:54:38.310226 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:38.310185 2568 generic.go:356] "Generic (PLEG): container finished" podID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerID="29bbf168314c04572731522ecbd364cdfd48b2f66a62c47ffc89224cfc0713b5" exitCode=0 Jul 06 12:54:38.310625 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:38.310255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" event={"ID":"2e41154f-5ed5-484b-99c2-b8b5c6c33511","Type":"ContainerDied","Data":"29bbf168314c04572731522ecbd364cdfd48b2f66a62c47ffc89224cfc0713b5"} Jul 06 12:54:39.437476 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.437452 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:39.541781 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.541749 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnkzf\" (UniqueName: \"kubernetes.io/projected/2e41154f-5ed5-484b-99c2-b8b5c6c33511-kube-api-access-dnkzf\") pod \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " Jul 06 12:54:39.541931 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.541793 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-util\") pod \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " Jul 06 12:54:39.541931 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.541826 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-bundle\") pod \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\" (UID: \"2e41154f-5ed5-484b-99c2-b8b5c6c33511\") " Jul 06 12:54:39.542574 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.542508 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-bundle" (OuterVolumeSpecName: "bundle") pod "2e41154f-5ed5-484b-99c2-b8b5c6c33511" (UID: "2e41154f-5ed5-484b-99c2-b8b5c6c33511"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:39.544115 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.544085 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e41154f-5ed5-484b-99c2-b8b5c6c33511-kube-api-access-dnkzf" (OuterVolumeSpecName: "kube-api-access-dnkzf") pod "2e41154f-5ed5-484b-99c2-b8b5c6c33511" (UID: "2e41154f-5ed5-484b-99c2-b8b5c6c33511"). InnerVolumeSpecName "kube-api-access-dnkzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:54:39.546192 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.546174 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-util" (OuterVolumeSpecName: "util") pod "2e41154f-5ed5-484b-99c2-b8b5c6c33511" (UID: "2e41154f-5ed5-484b-99c2-b8b5c6c33511"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:39.642502 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.642480 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkzf\" (UniqueName: \"kubernetes.io/projected/2e41154f-5ed5-484b-99c2-b8b5c6c33511-kube-api-access-dnkzf\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:54:39.642502 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.642499 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-util\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:54:39.642662 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:39.642509 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e41154f-5ed5-484b-99c2-b8b5c6c33511-bundle\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:54:40.318749 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:40.318725 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" Jul 06 12:54:40.318749 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:40.318733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g8prx" event={"ID":"2e41154f-5ed5-484b-99c2-b8b5c6c33511","Type":"ContainerDied","Data":"e2ac7f7eb12f9df286ac05c0a0a0d939e32accd0c46174e94e3c3af9ad25d595"} Jul 06 12:54:40.318994 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:40.318781 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ac7f7eb12f9df286ac05c0a0a0d939e32accd0c46174e94e3c3af9ad25d595" Jul 06 12:54:53.559875 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.559844 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6"] Jul 06 12:54:53.560341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.560109 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerName="pull" Jul 06 12:54:53.560341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.560120 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerName="pull" Jul 06 12:54:53.560341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.560139 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerName="util" Jul 06 12:54:53.560341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.560144 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerName="util" Jul 06 12:54:53.560341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.560152 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerName="extract" Jul 06 12:54:53.560341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.560158 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerName="extract" Jul 06 12:54:53.560341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.560223 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e41154f-5ed5-484b-99c2-b8b5c6c33511" containerName="extract" Jul 06 12:54:53.563253 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.563236 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.568443 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.568419 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 12:54:53.568662 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.568546 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 12:54:53.569678 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.569437 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2pj8k\"" Jul 06 12:54:53.601696 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.601669 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6"] Jul 06 12:54:53.638107 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.638080 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtq6\" (UniqueName: \"kubernetes.io/projected/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-kube-api-access-hrtq6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.638219 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.638123 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.638219 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.638189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.738984 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.738948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.739152 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.738999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.739152 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.739051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtq6\" (UniqueName: \"kubernetes.io/projected/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-kube-api-access-hrtq6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.739423 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.739402 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.739496 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.739431 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.763966 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.763937 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtq6\" (UniqueName: \"kubernetes.io/projected/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-kube-api-access-hrtq6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.871809 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.871786 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:53.899296 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.899272 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hkxl2"] Jul 06 12:54:53.902525 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.902508 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:53.917358 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.917338 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 12:54:53.922809 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.922581 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gj7jv\"" Jul 06 12:54:53.922940 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.922928 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 12:54:53.924997 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:53.924956 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hkxl2"] Jul 06 12:54:54.015546 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.015524 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6"] Jul 06 12:54:54.018076 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:54:54.018049 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f15ce8_b7fc_48ff_9dd8_fa2c4e9dab0b.slice/crio-1ca600c346edb02c9c373c274f4473d6ff35cd81aed7d40f6438bda7e1d646e6 WatchSource:0}: Error finding container 1ca600c346edb02c9c373c274f4473d6ff35cd81aed7d40f6438bda7e1d646e6: Status 404 returned error can't find the container with id 1ca600c346edb02c9c373c274f4473d6ff35cd81aed7d40f6438bda7e1d646e6 Jul 06 12:54:54.042299 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.042272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdzr\" (UniqueName: \"kubernetes.io/projected/c8b4bd89-5f31-47fb-9118-1091443f6427-kube-api-access-crdzr\") pod \"cert-manager-webhook-6486ffd997-hkxl2\" (UID: \"c8b4bd89-5f31-47fb-9118-1091443f6427\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:54.042396 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.042346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8b4bd89-5f31-47fb-9118-1091443f6427-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hkxl2\" (UID: \"c8b4bd89-5f31-47fb-9118-1091443f6427\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:54.143411 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.143352 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8b4bd89-5f31-47fb-9118-1091443f6427-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hkxl2\" (UID: \"c8b4bd89-5f31-47fb-9118-1091443f6427\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:54.143522 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.143439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdzr\" (UniqueName: \"kubernetes.io/projected/c8b4bd89-5f31-47fb-9118-1091443f6427-kube-api-access-crdzr\") pod \"cert-manager-webhook-6486ffd997-hkxl2\" (UID: \"c8b4bd89-5f31-47fb-9118-1091443f6427\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:54.153698 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.153673 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdzr\" (UniqueName: \"kubernetes.io/projected/c8b4bd89-5f31-47fb-9118-1091443f6427-kube-api-access-crdzr\") pod \"cert-manager-webhook-6486ffd997-hkxl2\" (UID: \"c8b4bd89-5f31-47fb-9118-1091443f6427\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:54.154006 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.153990 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8b4bd89-5f31-47fb-9118-1091443f6427-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hkxl2\" (UID: \"c8b4bd89-5f31-47fb-9118-1091443f6427\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:54.211171 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.211148 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:54:54.339173 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.339143 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hkxl2"] Jul 06 12:54:54.341970 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:54:54.341948 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8b4bd89_5f31_47fb_9118_1091443f6427.slice/crio-fe2e2171318789cfc2ed5b62ee71127aaf7f17f76357179435ff3a672da3685a WatchSource:0}: Error finding container fe2e2171318789cfc2ed5b62ee71127aaf7f17f76357179435ff3a672da3685a: Status 404 returned error can't find the container with id fe2e2171318789cfc2ed5b62ee71127aaf7f17f76357179435ff3a672da3685a Jul 06 12:54:54.369583 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.369548 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" event={"ID":"c8b4bd89-5f31-47fb-9118-1091443f6427","Type":"ContainerStarted","Data":"fe2e2171318789cfc2ed5b62ee71127aaf7f17f76357179435ff3a672da3685a"} Jul 06 12:54:54.371031 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.371009 2568 generic.go:356] "Generic (PLEG): container finished" podID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerID="1a1eebe28c53b0542dbf44ae2d11212ff84bf30aa6c3679d47f88a6148deafbf" exitCode=0 Jul 06 12:54:54.371136 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.371064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" event={"ID":"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b","Type":"ContainerDied","Data":"1a1eebe28c53b0542dbf44ae2d11212ff84bf30aa6c3679d47f88a6148deafbf"} Jul 06 12:54:54.371136 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:54.371093 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" event={"ID":"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b","Type":"ContainerStarted","Data":"1ca600c346edb02c9c373c274f4473d6ff35cd81aed7d40f6438bda7e1d646e6"} Jul 06 12:54:56.380761 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:56.380733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" event={"ID":"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b","Type":"ContainerStarted","Data":"97b3be229ed6a7a0b801edcee6c1bd454c884e90fde32c2469ae4e73eaaf4507"} Jul 06 12:54:57.386131 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:57.386099 2568 generic.go:356] "Generic (PLEG): container finished" podID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerID="97b3be229ed6a7a0b801edcee6c1bd454c884e90fde32c2469ae4e73eaaf4507" exitCode=0 Jul 06 12:54:57.386508 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:57.386187 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" event={"ID":"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b","Type":"ContainerDied","Data":"97b3be229ed6a7a0b801edcee6c1bd454c884e90fde32c2469ae4e73eaaf4507"} Jul 06 12:54:58.392440 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:58.392407 2568 generic.go:356] "Generic (PLEG): container finished" podID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerID="ff75192da2eec439ecd07ddcda631af27edfafdf845c883e95d172a185178d46" exitCode=0 Jul 06 12:54:58.392845 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:58.392473 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" event={"ID":"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b","Type":"ContainerDied","Data":"ff75192da2eec439ecd07ddcda631af27edfafdf845c883e95d172a185178d46"} Jul 06 12:54:59.529378 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.529353 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:54:59.688832 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.688754 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-util\") pod \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " Jul 06 12:54:59.688832 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.688796 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrtq6\" (UniqueName: \"kubernetes.io/projected/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-kube-api-access-hrtq6\") pod \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " Jul 06 12:54:59.689015 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.688858 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-bundle\") pod \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\" (UID: \"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b\") " Jul 06 12:54:59.689257 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.689229 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-bundle" (OuterVolumeSpecName: "bundle") pod "29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" (UID: "29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:54:59.691042 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.691021 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-kube-api-access-hrtq6" (OuterVolumeSpecName: "kube-api-access-hrtq6") pod "29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" (UID: "29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b"). InnerVolumeSpecName "kube-api-access-hrtq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:54:59.790149 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.790121 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrtq6\" (UniqueName: \"kubernetes.io/projected/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-kube-api-access-hrtq6\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:54:59.790149 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:54:59.790142 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-bundle\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:55:00.151985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:00.151918 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-util" (OuterVolumeSpecName: "util") pod "29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" (UID: "29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:55:00.193191 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:00.193166 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b-util\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:55:00.400726 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:00.400695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" event={"ID":"29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b","Type":"ContainerDied","Data":"1ca600c346edb02c9c373c274f4473d6ff35cd81aed7d40f6438bda7e1d646e6"} Jul 06 12:55:00.400726 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:00.400722 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f569r6" Jul 06 12:55:00.400915 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:00.400746 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ca600c346edb02c9c373c274f4473d6ff35cd81aed7d40f6438bda7e1d646e6" Jul 06 12:55:03.592307 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592263 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jnn2t"] Jul 06 12:55:03.592717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592540 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerName="util" Jul 06 12:55:03.592717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592550 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerName="util" Jul 06 12:55:03.592717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592558 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerName="extract" Jul 06 12:55:03.592717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592563 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerName="extract" Jul 06 12:55:03.592717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592579 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerName="pull" Jul 06 12:55:03.592717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592584 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerName="pull" Jul 06 12:55:03.592717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.592648 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f15ce8-b7fc-48ff-9dd8-fa2c4e9dab0b" containerName="extract" Jul 06 12:55:03.595400 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.595385 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:03.598010 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.597988 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-w5958\"" Jul 06 12:55:03.607676 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.607619 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jnn2t"] Jul 06 12:55:03.717969 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.717933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbt5\" (UniqueName: \"kubernetes.io/projected/2a7b572d-d6a9-4d14-95e2-4d184cbc13d5-kube-api-access-rtbt5\") pod \"cert-manager-55b4d954d5-jnn2t\" (UID: \"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5\") " pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:03.718147 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.717980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a7b572d-d6a9-4d14-95e2-4d184cbc13d5-bound-sa-token\") pod \"cert-manager-55b4d954d5-jnn2t\" (UID: \"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5\") " pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:03.818545 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.818507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbt5\" (UniqueName: \"kubernetes.io/projected/2a7b572d-d6a9-4d14-95e2-4d184cbc13d5-kube-api-access-rtbt5\") pod \"cert-manager-55b4d954d5-jnn2t\" (UID: \"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5\") " pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:03.818744 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.818556 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a7b572d-d6a9-4d14-95e2-4d184cbc13d5-bound-sa-token\") pod \"cert-manager-55b4d954d5-jnn2t\" (UID: \"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5\") " pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:03.832866 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.832834 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a7b572d-d6a9-4d14-95e2-4d184cbc13d5-bound-sa-token\") pod \"cert-manager-55b4d954d5-jnn2t\" (UID: \"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5\") " pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:03.832998 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.832913 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbt5\" (UniqueName: \"kubernetes.io/projected/2a7b572d-d6a9-4d14-95e2-4d184cbc13d5-kube-api-access-rtbt5\") pod \"cert-manager-55b4d954d5-jnn2t\" (UID: \"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5\") " pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:03.904040 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:03.903993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jnn2t" Jul 06 12:55:04.042645 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:04.042592 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jnn2t"] Jul 06 12:55:04.045306 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:55:04.045278 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a7b572d_d6a9_4d14_95e2_4d184cbc13d5.slice/crio-a1d09c3088faedfe4357b1b9eb92111b3da8d62af8695ee58d58db0d1059dad6 WatchSource:0}: Error finding container a1d09c3088faedfe4357b1b9eb92111b3da8d62af8695ee58d58db0d1059dad6: Status 404 returned error can't find the container with id a1d09c3088faedfe4357b1b9eb92111b3da8d62af8695ee58d58db0d1059dad6 Jul 06 12:55:04.415926 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:04.415885 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jnn2t" event={"ID":"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5","Type":"ContainerStarted","Data":"a1d09c3088faedfe4357b1b9eb92111b3da8d62af8695ee58d58db0d1059dad6"} Jul 06 12:55:07.430144 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:07.430102 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jnn2t" event={"ID":"2a7b572d-d6a9-4d14-95e2-4d184cbc13d5","Type":"ContainerStarted","Data":"b7720a41ef8c4421f862af8bafb4ebb288c117e682bc145b7c4c8e08d2c5962d"} Jul 06 12:55:07.431658 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:07.431616 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" event={"ID":"c8b4bd89-5f31-47fb-9118-1091443f6427","Type":"ContainerStarted","Data":"ca4a1646d2176e2e5954ca24760889f9c1ffcd6e959df917e9119447ca9b2c6e"} Jul 06 12:55:07.431777 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:07.431751 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:55:07.456947 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:07.456897 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jnn2t" podStartSLOduration=1.466346422 podStartE2EDuration="4.456882324s" totalImagesPullingTime="2.990535902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:55:07.454483854 +0000 UTC m=+491.395324713" watchObservedRunningTime="2026-07-06 12:55:07.456882324 +0000 UTC m=+491.397723183" Jul 06 12:55:07.478884 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:07.478834 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" podStartSLOduration=1.778719521 podStartE2EDuration="14.478818616s" totalImagesPullingTime="12.700099095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:55:07.47727634 +0000 UTC m=+491.418117199" watchObservedRunningTime="2026-07-06 12:55:07.478818616 +0000 UTC m=+491.419659474" Jul 06 12:55:11.297854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.297817 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk"] Jul 06 12:55:11.314390 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.314345 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk"] Jul 06 12:55:11.314571 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.314471 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.317324 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.317291 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 12:55:11.318324 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.318297 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2pj8k\"" Jul 06 12:55:11.318544 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.318373 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 12:55:11.379244 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.379207 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.379416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.379271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r54k\" (UniqueName: \"kubernetes.io/projected/a0f039be-32f0-4345-8882-26f2ed93be05-kube-api-access-5r54k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.379416 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.379297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.480477 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.480438 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.480666 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.480503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r54k\" (UniqueName: \"kubernetes.io/projected/a0f039be-32f0-4345-8882-26f2ed93be05-kube-api-access-5r54k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.480666 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.480528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.480857 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.480840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.480897 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.480866 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.489988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.489957 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r54k\" (UniqueName: \"kubernetes.io/projected/a0f039be-32f0-4345-8882-26f2ed93be05-kube-api-access-5r54k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.624792 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.624767 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:11.770904 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:11.768473 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk"] Jul 06 12:55:11.772311 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:55:11.772287 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f039be_32f0_4345_8882_26f2ed93be05.slice/crio-8c973b8c0d7ce7d3799fa230fa585f8dc8c5eeac388f2e8d718c0a09efea9103 WatchSource:0}: Error finding container 8c973b8c0d7ce7d3799fa230fa585f8dc8c5eeac388f2e8d718c0a09efea9103: Status 404 returned error can't find the container with id 8c973b8c0d7ce7d3799fa230fa585f8dc8c5eeac388f2e8d718c0a09efea9103 Jul 06 12:55:12.451610 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:12.451577 2568 generic.go:356] "Generic (PLEG): container finished" podID="a0f039be-32f0-4345-8882-26f2ed93be05" containerID="219b34d6996a8533263498f1ee6ffb3de77bc673bf903bcbf18154847717c2f7" exitCode=0 Jul 06 12:55:12.452008 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:12.451672 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" event={"ID":"a0f039be-32f0-4345-8882-26f2ed93be05","Type":"ContainerDied","Data":"219b34d6996a8533263498f1ee6ffb3de77bc673bf903bcbf18154847717c2f7"} Jul 06 12:55:12.452008 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:12.451722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" event={"ID":"a0f039be-32f0-4345-8882-26f2ed93be05","Type":"ContainerStarted","Data":"8c973b8c0d7ce7d3799fa230fa585f8dc8c5eeac388f2e8d718c0a09efea9103"} Jul 06 12:55:13.437949 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:13.437919 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hkxl2" Jul 06 12:55:14.461965 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:14.461932 2568 generic.go:356] "Generic (PLEG): container finished" podID="a0f039be-32f0-4345-8882-26f2ed93be05" containerID="3c0714eb61cb5bf61b507c9da323ff76731adb557ff96b1041a57d2a3cd1682e" exitCode=0 Jul 06 12:55:14.462436 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:14.461994 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" event={"ID":"a0f039be-32f0-4345-8882-26f2ed93be05","Type":"ContainerDied","Data":"3c0714eb61cb5bf61b507c9da323ff76731adb557ff96b1041a57d2a3cd1682e"} Jul 06 12:55:15.467341 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:15.467302 2568 generic.go:356] "Generic (PLEG): container finished" podID="a0f039be-32f0-4345-8882-26f2ed93be05" containerID="e2c4d0882b7739920f82f37f96006927abc43d6571e492f228864a45dc94cbfb" exitCode=0 Jul 06 12:55:15.467746 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:15.467365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" event={"ID":"a0f039be-32f0-4345-8882-26f2ed93be05","Type":"ContainerDied","Data":"e2c4d0882b7739920f82f37f96006927abc43d6571e492f228864a45dc94cbfb"} Jul 06 12:55:16.594223 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.594199 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:16.619890 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.619868 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-util\") pod \"a0f039be-32f0-4345-8882-26f2ed93be05\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " Jul 06 12:55:16.620011 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.619925 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-bundle\") pod \"a0f039be-32f0-4345-8882-26f2ed93be05\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " Jul 06 12:55:16.620011 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.619984 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r54k\" (UniqueName: \"kubernetes.io/projected/a0f039be-32f0-4345-8882-26f2ed93be05-kube-api-access-5r54k\") pod \"a0f039be-32f0-4345-8882-26f2ed93be05\" (UID: \"a0f039be-32f0-4345-8882-26f2ed93be05\") " Jul 06 12:55:16.620775 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.620680 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-bundle" (OuterVolumeSpecName: "bundle") pod "a0f039be-32f0-4345-8882-26f2ed93be05" (UID: "a0f039be-32f0-4345-8882-26f2ed93be05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:55:16.622170 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.622147 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f039be-32f0-4345-8882-26f2ed93be05-kube-api-access-5r54k" (OuterVolumeSpecName: "kube-api-access-5r54k") pod "a0f039be-32f0-4345-8882-26f2ed93be05" (UID: "a0f039be-32f0-4345-8882-26f2ed93be05"). InnerVolumeSpecName "kube-api-access-5r54k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:55:16.627309 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.627285 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-util" (OuterVolumeSpecName: "util") pod "a0f039be-32f0-4345-8882-26f2ed93be05" (UID: "a0f039be-32f0-4345-8882-26f2ed93be05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:55:16.720959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.720929 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r54k\" (UniqueName: \"kubernetes.io/projected/a0f039be-32f0-4345-8882-26f2ed93be05-kube-api-access-5r54k\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:55:16.720959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.720954 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-util\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:55:16.720959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:16.720964 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f039be-32f0-4345-8882-26f2ed93be05-bundle\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:55:17.478009 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:17.477981 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" Jul 06 12:55:17.478009 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:17.477988 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6pcqk" event={"ID":"a0f039be-32f0-4345-8882-26f2ed93be05","Type":"ContainerDied","Data":"8c973b8c0d7ce7d3799fa230fa585f8dc8c5eeac388f2e8d718c0a09efea9103"} Jul 06 12:55:17.478211 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:17.478028 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c973b8c0d7ce7d3799fa230fa585f8dc8c5eeac388f2e8d718c0a09efea9103" Jul 06 12:55:24.455933 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.455888 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj"] Jul 06 12:55:24.456428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.456290 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f039be-32f0-4345-8882-26f2ed93be05" containerName="util" Jul 06 12:55:24.456428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.456304 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f039be-32f0-4345-8882-26f2ed93be05" containerName="util" Jul 06 12:55:24.456428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.456316 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f039be-32f0-4345-8882-26f2ed93be05" containerName="extract" Jul 06 12:55:24.456428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.456325 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f039be-32f0-4345-8882-26f2ed93be05" containerName="extract" Jul 06 12:55:24.456428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.456354 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f039be-32f0-4345-8882-26f2ed93be05" containerName="pull" Jul 06 12:55:24.456428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.456362 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f039be-32f0-4345-8882-26f2ed93be05" containerName="pull" Jul 06 12:55:24.456428 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.456429 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f039be-32f0-4345-8882-26f2ed93be05" containerName="extract" Jul 06 12:55:24.459833 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.459805 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.464541 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.464519 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 12:55:24.465367 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.465350 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 12:55:24.466037 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.465968 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 12:55:24.466037 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.466027 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t7n7s\"" Jul 06 12:55:24.466215 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.466027 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 12:55:24.476842 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.476821 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9a8651c-19b3-44f2-9466-b217500b3b8c-tmp\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.476928 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.476854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9a8651c-19b3-44f2-9466-b217500b3b8c-webhook-cert\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.476928 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.476918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9a8651c-19b3-44f2-9466-b217500b3b8c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.477013 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.476970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9a8651c-19b3-44f2-9466-b217500b3b8c-manifests\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.477013 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.476987 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk22b\" (UniqueName: \"kubernetes.io/projected/d9a8651c-19b3-44f2-9466-b217500b3b8c-kube-api-access-fk22b\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.495512 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.495471 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj"] Jul 06 12:55:24.578220 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.578183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9a8651c-19b3-44f2-9466-b217500b3b8c-webhook-cert\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.578402 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.578230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9a8651c-19b3-44f2-9466-b217500b3b8c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.578402 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.578256 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9a8651c-19b3-44f2-9466-b217500b3b8c-manifests\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.578402 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.578272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk22b\" (UniqueName: \"kubernetes.io/projected/d9a8651c-19b3-44f2-9466-b217500b3b8c-kube-api-access-fk22b\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.578402 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.578291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9a8651c-19b3-44f2-9466-b217500b3b8c-tmp\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.578776 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.578747 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9a8651c-19b3-44f2-9466-b217500b3b8c-manifests\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.580659 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.580615 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9a8651c-19b3-44f2-9466-b217500b3b8c-tmp\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.580759 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.580678 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9a8651c-19b3-44f2-9466-b217500b3b8c-webhook-cert\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.580759 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.580746 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9a8651c-19b3-44f2-9466-b217500b3b8c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.592689 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.592664 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk22b\" (UniqueName: \"kubernetes.io/projected/d9a8651c-19b3-44f2-9466-b217500b3b8c-kube-api-access-fk22b\") pod \"opendatahub-operator-controller-manager-58899bc45d-zc6fj\" (UID: \"d9a8651c-19b3-44f2-9466-b217500b3b8c\") " pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.771407 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.771320 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:24.938330 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:24.938299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj"] Jul 06 12:55:24.940855 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:55:24.940831 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a8651c_19b3_44f2_9466_b217500b3b8c.slice/crio-ed76a416cd79336394f79fee21dce66b5c9d18eab46ffed78d7caaf8774563d7 WatchSource:0}: Error finding container ed76a416cd79336394f79fee21dce66b5c9d18eab46ffed78d7caaf8774563d7: Status 404 returned error can't find the container with id ed76a416cd79336394f79fee21dce66b5c9d18eab46ffed78d7caaf8774563d7 Jul 06 12:55:25.512860 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:25.512823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" event={"ID":"d9a8651c-19b3-44f2-9466-b217500b3b8c","Type":"ContainerStarted","Data":"ed76a416cd79336394f79fee21dce66b5c9d18eab46ffed78d7caaf8774563d7"} Jul 06 12:55:28.530122 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:28.530088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" event={"ID":"d9a8651c-19b3-44f2-9466-b217500b3b8c","Type":"ContainerStarted","Data":"bf0ac218ddc0bd95782c8e789eec835ecf1cee9afa80d87f6a977be84147945a"} Jul 06 12:55:29.536015 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:29.535972 2568 generic.go:356] "Generic (PLEG): container finished" podID="d9a8651c-19b3-44f2-9466-b217500b3b8c" containerID="bf0ac218ddc0bd95782c8e789eec835ecf1cee9afa80d87f6a977be84147945a" exitCode=0 Jul 06 12:55:29.536398 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:29.536013 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" event={"ID":"d9a8651c-19b3-44f2-9466-b217500b3b8c","Type":"ContainerDied","Data":"bf0ac218ddc0bd95782c8e789eec835ecf1cee9afa80d87f6a977be84147945a"} Jul 06 12:55:30.542148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:30.542115 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" event={"ID":"d9a8651c-19b3-44f2-9466-b217500b3b8c","Type":"ContainerStarted","Data":"3e49fafb657746b1596cb7ca2d028b0be2b7982359ffb2d417b756ca715b9eb4"} Jul 06 12:55:30.542613 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:30.542209 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:30.583413 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:30.583347 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" podStartSLOduration=2.229503754 podStartE2EDuration="6.58332648s" totalImagesPullingTime="3.328836411s" totalInitContainerRuntime="1.024986315s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:55:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:55:30.582788483 +0000 UTC m=+514.523629378" watchObservedRunningTime="2026-07-06 12:55:30.58332648 +0000 UTC m=+514.524167341" Jul 06 12:55:41.549593 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:41.549560 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58899bc45d-zc6fj" Jul 06 12:55:45.019156 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.019118 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh"] Jul 06 12:55:45.026260 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.026236 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.031923 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.031901 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 12:55:45.032591 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.032567 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 12:55:45.032717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.032569 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x9rtj\"" Jul 06 12:55:45.033109 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.033091 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 12:55:45.033191 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.033159 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 12:55:45.037138 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.037121 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 12:55:45.050304 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.050283 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh"] Jul 06 12:55:45.146944 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.146914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-cert\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.147106 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.146968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-manager-config\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.147106 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.146997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-metrics-cert\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.147106 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.147018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4hj\" (UniqueName: \"kubernetes.io/projected/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-kube-api-access-nb4hj\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.247779 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.247751 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-cert\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.248057 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.247807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-manager-config\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.248057 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.247837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-metrics-cert\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.248057 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.247857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4hj\" (UniqueName: \"kubernetes.io/projected/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-kube-api-access-nb4hj\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.248812 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.248779 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-manager-config\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.251056 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.251036 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-cert\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.251220 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.251197 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-metrics-cert\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.268401 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.268378 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4hj\" (UniqueName: \"kubernetes.io/projected/fa14f5a6-3e90-41f2-9e4b-48d142f1dc06-kube-api-access-nb4hj\") pod \"lws-controller-manager-67468f6576-rkbwh\" (UID: \"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06\") " pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.335468 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.335377 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:45.496277 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.496235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh"] Jul 06 12:55:45.497060 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:55:45.497036 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa14f5a6_3e90_41f2_9e4b_48d142f1dc06.slice/crio-a8fe6df204580ab5d2d6b6e0f2e18c5aa91b3c0c11619a01bad68e7b320c2b68 WatchSource:0}: Error finding container a8fe6df204580ab5d2d6b6e0f2e18c5aa91b3c0c11619a01bad68e7b320c2b68: Status 404 returned error can't find the container with id a8fe6df204580ab5d2d6b6e0f2e18c5aa91b3c0c11619a01bad68e7b320c2b68 Jul 06 12:55:45.611475 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:45.611442 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" event={"ID":"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06","Type":"ContainerStarted","Data":"a8fe6df204580ab5d2d6b6e0f2e18c5aa91b3c0c11619a01bad68e7b320c2b68"} Jul 06 12:55:47.622870 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:47.622832 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" event={"ID":"fa14f5a6-3e90-41f2-9e4b-48d142f1dc06","Type":"ContainerStarted","Data":"7ce6a86a8d55e4ef8d81d1c24696c062bf487870db73d848010c7b6c685531d8"} Jul 06 12:55:47.623257 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:47.622905 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:55:47.658180 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:47.658013 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" podStartSLOduration=1.811481192 podStartE2EDuration="3.657994529s" totalImagesPullingTime="1.846513337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:55:47.655985855 +0000 UTC m=+531.596826715" watchObservedRunningTime="2026-07-06 12:55:47.657994529 +0000 UTC m=+531.598835389" Jul 06 12:55:58.629354 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:55:58.629330 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67468f6576-rkbwh" Jul 06 12:56:08.617475 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.617438 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m"] Jul 06 12:56:08.624619 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.624597 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.632959 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.632929 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 12:56:08.633085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.632966 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2lntl\"" Jul 06 12:56:08.633085 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.633025 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 12:56:08.635186 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.635170 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 12:56:08.661671 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.661647 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m"] Jul 06 12:56:08.747709 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-token\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.747709 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.747912 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zh2d\" (UniqueName: \"kubernetes.io/projected/08f5adc1-7670-4b9a-9619-f8add9ce4a79-kube-api-access-7zh2d\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.747912 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747805 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-data\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.747912 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.747912 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.748039 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747919 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.748039 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.748039 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.747964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-data\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849284 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849284 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849284 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849284 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849284 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849274 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849521 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-token\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849521 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849622 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849512 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-data\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849622 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zh2d\" (UniqueName: \"kubernetes.io/projected/08f5adc1-7670-4b9a-9619-f8add9ce4a79-kube-api-access-7zh2d\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849622 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849583 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849871 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849848 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849923 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849889 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.849923 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.849913 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.851917 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.851881 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.852118 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.852099 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.891046 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.890974 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f5adc1-7670-4b9a-9619-f8add9ce4a79-istio-token\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.910469 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.910443 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zh2d\" (UniqueName: \"kubernetes.io/projected/08f5adc1-7670-4b9a-9619-f8add9ce4a79-kube-api-access-7zh2d\") pod \"data-science-gateway-data-science-gateway-class-656c84dccdvvk4m\" (UID: \"08f5adc1-7670-4b9a-9619-f8add9ce4a79\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:08.934386 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:08.934360 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:09.159135 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:09.159056 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m"] Jul 06 12:56:09.161869 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:56:09.161845 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f5adc1_7670_4b9a_9619_f8add9ce4a79.slice/crio-66c2148667617ff5c06ec182e8f70e1c493590aead637e9868a47e98edbec298 WatchSource:0}: Error finding container 66c2148667617ff5c06ec182e8f70e1c493590aead637e9868a47e98edbec298: Status 404 returned error can't find the container with id 66c2148667617ff5c06ec182e8f70e1c493590aead637e9868a47e98edbec298 Jul 06 12:56:09.719526 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:09.719492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" event={"ID":"08f5adc1-7670-4b9a-9619-f8add9ce4a79","Type":"ContainerStarted","Data":"66c2148667617ff5c06ec182e8f70e1c493590aead637e9868a47e98edbec298"} Jul 06 12:56:11.694289 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:11.694236 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:56:11.694597 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:11.694339 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:56:11.694597 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:11.694376 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:56:12.734792 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:12.734751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" event={"ID":"08f5adc1-7670-4b9a-9619-f8add9ce4a79","Type":"ContainerStarted","Data":"4a3eac82508f1c13874984af4d77a80d21ed7f6900e317bd56600107ff61bd00"} Jul 06 12:56:12.772652 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:12.772585 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" podStartSLOduration=2.242497395 podStartE2EDuration="4.772566125s" totalImagesPullingTime="2.53006873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:56:12.771093319 +0000 UTC m=+556.711934205" watchObservedRunningTime="2026-07-06 12:56:12.772566125 +0000 UTC m=+556.713406986" Jul 06 12:56:12.943394 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:12.943356 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:12.948236 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:12.948206 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:13.739615 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:13.739575 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:13.740463 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:13.740446 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-656c84dccdvvk4m" Jul 06 12:56:45.850089 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:45.850054 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b7d49"] Jul 06 12:56:45.860007 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:45.859985 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" Jul 06 12:56:45.866899 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:45.866585 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 12:56:45.867462 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:45.867443 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 12:56:45.867578 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:45.867446 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-fzql7\"" Jul 06 12:56:45.895662 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:45.895606 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b7d49"] Jul 06 12:56:45.928409 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:45.928377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8fj9\" (UniqueName: \"kubernetes.io/projected/9848eeb1-3652-419e-a540-4ad746c0cf3f-kube-api-access-v8fj9\") pod \"kuadrant-operator-catalog-b7d49\" (UID: \"9848eeb1-3652-419e-a540-4ad746c0cf3f\") " pod="kuadrant-system/kuadrant-operator-catalog-b7d49" Jul 06 12:56:46.029337 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.029305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8fj9\" (UniqueName: \"kubernetes.io/projected/9848eeb1-3652-419e-a540-4ad746c0cf3f-kube-api-access-v8fj9\") pod \"kuadrant-operator-catalog-b7d49\" (UID: \"9848eeb1-3652-419e-a540-4ad746c0cf3f\") " pod="kuadrant-system/kuadrant-operator-catalog-b7d49" Jul 06 12:56:46.040558 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.040532 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8fj9\" (UniqueName: \"kubernetes.io/projected/9848eeb1-3652-419e-a540-4ad746c0cf3f-kube-api-access-v8fj9\") pod \"kuadrant-operator-catalog-b7d49\" (UID: \"9848eeb1-3652-419e-a540-4ad746c0cf3f\") " pod="kuadrant-system/kuadrant-operator-catalog-b7d49" Jul 06 12:56:46.159075 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.158986 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b7d49"] Jul 06 12:56:46.159314 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.159288 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" Jul 06 12:56:46.300894 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.300867 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b7d49"] Jul 06 12:56:46.303457 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:56:46.303431 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9848eeb1_3652_419e_a540_4ad746c0cf3f.slice/crio-994adc134217d488dbdd9c3a82da4f569bc08b8f43248aadd6f98b62f38de224 WatchSource:0}: Error finding container 994adc134217d488dbdd9c3a82da4f569bc08b8f43248aadd6f98b62f38de224: Status 404 returned error can't find the container with id 994adc134217d488dbdd9c3a82da4f569bc08b8f43248aadd6f98b62f38de224 Jul 06 12:56:46.394589 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.394556 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t442n"] Jul 06 12:56:46.399452 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.399430 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:46.432335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.432282 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t442n"] Jul 06 12:56:46.433610 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.433592 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7mx\" (UniqueName: \"kubernetes.io/projected/b00e504e-6271-40ac-8dd4-7445da038b47-kube-api-access-4d7mx\") pod \"kuadrant-operator-catalog-t442n\" (UID: \"b00e504e-6271-40ac-8dd4-7445da038b47\") " pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:46.534602 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.534575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7mx\" (UniqueName: \"kubernetes.io/projected/b00e504e-6271-40ac-8dd4-7445da038b47-kube-api-access-4d7mx\") pod \"kuadrant-operator-catalog-t442n\" (UID: \"b00e504e-6271-40ac-8dd4-7445da038b47\") " pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:46.551154 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.551128 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7mx\" (UniqueName: \"kubernetes.io/projected/b00e504e-6271-40ac-8dd4-7445da038b47-kube-api-access-4d7mx\") pod \"kuadrant-operator-catalog-t442n\" (UID: \"b00e504e-6271-40ac-8dd4-7445da038b47\") " pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:46.709007 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.708946 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:46.876973 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.876782 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t442n"] Jul 06 12:56:46.884530 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:46.884472 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" event={"ID":"9848eeb1-3652-419e-a540-4ad746c0cf3f","Type":"ContainerStarted","Data":"994adc134217d488dbdd9c3a82da4f569bc08b8f43248aadd6f98b62f38de224"} Jul 06 12:56:46.938356 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:56:46.938318 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00e504e_6271_40ac_8dd4_7445da038b47.slice/crio-70609fc2214ee406f76aaf0a424bfc1202176e81e186368ddd60f2f59d4cc9a3 WatchSource:0}: Error finding container 70609fc2214ee406f76aaf0a424bfc1202176e81e186368ddd60f2f59d4cc9a3: Status 404 returned error can't find the container with id 70609fc2214ee406f76aaf0a424bfc1202176e81e186368ddd60f2f59d4cc9a3 Jul 06 12:56:47.890912 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:47.890868 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t442n" event={"ID":"b00e504e-6271-40ac-8dd4-7445da038b47","Type":"ContainerStarted","Data":"70609fc2214ee406f76aaf0a424bfc1202176e81e186368ddd60f2f59d4cc9a3"} Jul 06 12:56:48.896430 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:48.896390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" event={"ID":"9848eeb1-3652-419e-a540-4ad746c0cf3f","Type":"ContainerStarted","Data":"2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984"} Jul 06 12:56:48.896936 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:48.896466 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" podUID="9848eeb1-3652-419e-a540-4ad746c0cf3f" containerName="registry-server" containerID="cri-o://2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984" gracePeriod=2 Jul 06 12:56:48.897983 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:48.897953 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t442n" event={"ID":"b00e504e-6271-40ac-8dd4-7445da038b47","Type":"ContainerStarted","Data":"899642d091743dc5ec726c83af6f5763c5fe8f472e1ae1c94b7ece14d235ac66"} Jul 06 12:56:48.921010 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:48.920962 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" podStartSLOduration=2.073377774 podStartE2EDuration="3.920945454s" totalImagesPullingTime="1.84756768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:56:48.919512799 +0000 UTC m=+592.860353658" watchObservedRunningTime="2026-07-06 12:56:48.920945454 +0000 UTC m=+592.861786309" Jul 06 12:56:48.939997 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:48.939936 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-t442n" podStartSLOduration=1.7263679010000001 podStartE2EDuration="2.939917365s" totalImagesPullingTime="1.213549464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:56:48.938543482 +0000 UTC m=+592.879384341" watchObservedRunningTime="2026-07-06 12:56:48.939917365 +0000 UTC m=+592.880758224" Jul 06 12:56:49.142404 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.142381 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" Jul 06 12:56:49.259470 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.259386 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8fj9\" (UniqueName: \"kubernetes.io/projected/9848eeb1-3652-419e-a540-4ad746c0cf3f-kube-api-access-v8fj9\") pod \"9848eeb1-3652-419e-a540-4ad746c0cf3f\" (UID: \"9848eeb1-3652-419e-a540-4ad746c0cf3f\") " Jul 06 12:56:49.261651 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.261613 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9848eeb1-3652-419e-a540-4ad746c0cf3f-kube-api-access-v8fj9" (OuterVolumeSpecName: "kube-api-access-v8fj9") pod "9848eeb1-3652-419e-a540-4ad746c0cf3f" (UID: "9848eeb1-3652-419e-a540-4ad746c0cf3f"). InnerVolumeSpecName "kube-api-access-v8fj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:56:49.360282 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.360257 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8fj9\" (UniqueName: \"kubernetes.io/projected/9848eeb1-3652-419e-a540-4ad746c0cf3f-kube-api-access-v8fj9\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:56:49.903716 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.903685 2568 generic.go:356] "Generic (PLEG): container finished" podID="9848eeb1-3652-419e-a540-4ad746c0cf3f" containerID="2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984" exitCode=0 Jul 06 12:56:49.904191 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.903729 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" Jul 06 12:56:49.904191 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.903770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" event={"ID":"9848eeb1-3652-419e-a540-4ad746c0cf3f","Type":"ContainerDied","Data":"2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984"} Jul 06 12:56:49.904191 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.903832 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b7d49" event={"ID":"9848eeb1-3652-419e-a540-4ad746c0cf3f","Type":"ContainerDied","Data":"994adc134217d488dbdd9c3a82da4f569bc08b8f43248aadd6f98b62f38de224"} Jul 06 12:56:49.904191 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.903875 2568 scope.go:117] "RemoveContainer" containerID="2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984" Jul 06 12:56:49.913054 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.913037 2568 scope.go:117] "RemoveContainer" containerID="2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984" Jul 06 12:56:49.913315 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:56:49.913299 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984\": container with ID starting with 2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984 not found: ID does not exist" containerID="2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984" Jul 06 12:56:49.913380 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.913324 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984"} err="failed to get container status \"2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984\": rpc error: code = NotFound desc = could not find container \"2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984\": container with ID starting with 2ea9c9910b6582e9009502d4e5673665cd00bb38b5c7f702687f7ee85df86984 not found: ID does not exist" Jul 06 12:56:49.928413 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.928390 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b7d49"] Jul 06 12:56:49.931870 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:49.931851 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b7d49"] Jul 06 12:56:50.590778 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:50.590747 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9848eeb1-3652-419e-a540-4ad746c0cf3f" path="/var/lib/kubelet/pods/9848eeb1-3652-419e-a540-4ad746c0cf3f/volumes" Jul 06 12:56:56.519215 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:56.519188 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:56:56.521534 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:56.521508 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 12:56:56.710076 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:56.710047 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:56.710237 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:56.710097 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:56.732108 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:56.732086 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:56.960620 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:56.960598 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-t442n" Jul 06 12:56:58.648155 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.648122 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69"] Jul 06 12:56:58.648523 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.648449 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9848eeb1-3652-419e-a540-4ad746c0cf3f" containerName="registry-server" Jul 06 12:56:58.648523 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.648460 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9848eeb1-3652-419e-a540-4ad746c0cf3f" containerName="registry-server" Jul 06 12:56:58.648598 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.648530 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9848eeb1-3652-419e-a540-4ad746c0cf3f" containerName="registry-server" Jul 06 12:56:58.652891 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.652874 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.656058 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.656041 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7v8gc\"" Jul 06 12:56:58.668101 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.668083 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69"] Jul 06 12:56:58.738388 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.738365 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.738502 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.738433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j67f\" (UniqueName: \"kubernetes.io/projected/dcdb7470-71d1-4292-b587-a09ff2e027fb-kube-api-access-5j67f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.738502 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.738464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.839590 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.839562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j67f\" (UniqueName: \"kubernetes.io/projected/dcdb7470-71d1-4292-b587-a09ff2e027fb-kube-api-access-5j67f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.839723 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.839596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.839723 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.839625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.839945 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.839927 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.839988 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.839969 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.849121 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.849101 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j67f\" (UniqueName: \"kubernetes.io/projected/dcdb7470-71d1-4292-b587-a09ff2e027fb-kube-api-access-5j67f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:58.962597 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:58.962519 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:56:59.097227 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.097127 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69"] Jul 06 12:56:59.099982 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:56:59.099956 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdb7470_71d1_4292_b587_a09ff2e027fb.slice/crio-dfa6780fefc510adc2b1db5f9bf7def12f2adc77519b006b84e0aea929c1215f WatchSource:0}: Error finding container dfa6780fefc510adc2b1db5f9bf7def12f2adc77519b006b84e0aea929c1215f: Status 404 returned error can't find the container with id dfa6780fefc510adc2b1db5f9bf7def12f2adc77519b006b84e0aea929c1215f Jul 06 12:56:59.229256 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.229200 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6"] Jul 06 12:56:59.232802 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.232784 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.241289 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.241263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6"] Jul 06 12:56:59.343717 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.343681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tb8\" (UniqueName: \"kubernetes.io/projected/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-kube-api-access-n6tb8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.343840 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.343764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.343840 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.343794 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.445090 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.445062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.445200 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.445097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.445200 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.445127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tb8\" (UniqueName: \"kubernetes.io/projected/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-kube-api-access-n6tb8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.445414 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.445396 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.445487 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.445467 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.454746 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.454722 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tb8\" (UniqueName: \"kubernetes.io/projected/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-kube-api-access-n6tb8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.575451 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.575380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:56:59.705399 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.705374 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6"] Jul 06 12:56:59.707264 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:56:59.707237 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31d65be_4bf7_45dc_ba7f_0fe99a0c6d0b.slice/crio-04d663151a6bc298ba6b5b7f39e5d81974601bd8053f4eb033fb3702e02c5a6e WatchSource:0}: Error finding container 04d663151a6bc298ba6b5b7f39e5d81974601bd8053f4eb033fb3702e02c5a6e: Status 404 returned error can't find the container with id 04d663151a6bc298ba6b5b7f39e5d81974601bd8053f4eb033fb3702e02c5a6e Jul 06 12:56:59.828914 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.828859 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m"] Jul 06 12:56:59.832151 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.832132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:56:59.842257 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.842235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m"] Jul 06 12:56:59.950471 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.950443 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:56:59.950583 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.950494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kqp\" (UniqueName: \"kubernetes.io/projected/9e665150-004d-4d22-a98a-40ffeb477e21-kube-api-access-c4kqp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:56:59.950583 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.950554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:56:59.952698 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.952672 2568 generic.go:356] "Generic (PLEG): container finished" podID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerID="a82961381dcd2c173e100a1a75b60fff7e8b67128441cbf728e8da32d50b4b9d" exitCode=0 Jul 06 12:56:59.952800 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.952755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" event={"ID":"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b","Type":"ContainerDied","Data":"a82961381dcd2c173e100a1a75b60fff7e8b67128441cbf728e8da32d50b4b9d"} Jul 06 12:56:59.952854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.952799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" event={"ID":"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b","Type":"ContainerStarted","Data":"04d663151a6bc298ba6b5b7f39e5d81974601bd8053f4eb033fb3702e02c5a6e"} Jul 06 12:56:59.954278 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.954260 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerID="ec695cd12a8b8f215d89bdbf8d976dd757234a71eff4d42addf878ca07eaf545" exitCode=0 Jul 06 12:56:59.954410 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.954325 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" event={"ID":"dcdb7470-71d1-4292-b587-a09ff2e027fb","Type":"ContainerDied","Data":"ec695cd12a8b8f215d89bdbf8d976dd757234a71eff4d42addf878ca07eaf545"} Jul 06 12:56:59.954410 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:56:59.954353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" event={"ID":"dcdb7470-71d1-4292-b587-a09ff2e027fb","Type":"ContainerStarted","Data":"dfa6780fefc510adc2b1db5f9bf7def12f2adc77519b006b84e0aea929c1215f"} Jul 06 12:57:00.051906 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.051854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:00.052096 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.051975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:00.052096 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.052003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kqp\" (UniqueName: \"kubernetes.io/projected/9e665150-004d-4d22-a98a-40ffeb477e21-kube-api-access-c4kqp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:00.052210 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.052160 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:00.052362 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.052344 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:00.061724 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.061695 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kqp\" (UniqueName: \"kubernetes.io/projected/9e665150-004d-4d22-a98a-40ffeb477e21-kube-api-access-c4kqp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:00.188806 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.188775 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:00.325534 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.325506 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m"] Jul 06 12:57:00.327429 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:00.327403 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e665150_004d_4d22_a98a_40ffeb477e21.slice/crio-e90144f04fe72580c44203e734f40027d5ef75b0ef807ae7608539e142080959 WatchSource:0}: Error finding container e90144f04fe72580c44203e734f40027d5ef75b0ef807ae7608539e142080959: Status 404 returned error can't find the container with id e90144f04fe72580c44203e734f40027d5ef75b0ef807ae7608539e142080959 Jul 06 12:57:00.431757 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.431731 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2"] Jul 06 12:57:00.435082 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.435065 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.447571 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.447515 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2"] Jul 06 12:57:00.556674 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.556649 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qtk\" (UniqueName: \"kubernetes.io/projected/f9100767-6c78-4bea-8152-c15731093106-kube-api-access-c9qtk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.556789 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.556724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.556789 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.556745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.657749 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.657722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.657927 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.657768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.657927 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.657825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qtk\" (UniqueName: \"kubernetes.io/projected/f9100767-6c78-4bea-8152-c15731093106-kube-api-access-c9qtk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.658149 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.658117 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.658218 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.658175 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.667474 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.667453 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qtk\" (UniqueName: \"kubernetes.io/projected/f9100767-6c78-4bea-8152-c15731093106-kube-api-access-c9qtk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.805198 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.805168 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:00.960470 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.960441 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2"] Jul 06 12:57:00.961187 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.961158 2568 generic.go:356] "Generic (PLEG): container finished" podID="9e665150-004d-4d22-a98a-40ffeb477e21" containerID="53e5553b785afd16494ef3ba7efe364df48a165f6989a075422a1abff19e6b2e" exitCode=0 Jul 06 12:57:00.961305 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.961248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" event={"ID":"9e665150-004d-4d22-a98a-40ffeb477e21","Type":"ContainerDied","Data":"53e5553b785afd16494ef3ba7efe364df48a165f6989a075422a1abff19e6b2e"} Jul 06 12:57:00.961305 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.961299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" event={"ID":"9e665150-004d-4d22-a98a-40ffeb477e21","Type":"ContainerStarted","Data":"e90144f04fe72580c44203e734f40027d5ef75b0ef807ae7608539e142080959"} Jul 06 12:57:00.962496 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:00.962104 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9100767_6c78_4bea_8152_c15731093106.slice/crio-230e79af9b25c5d24378cca9db43270686e309b63ea2a3084fa3b059154a7c99 WatchSource:0}: Error finding container 230e79af9b25c5d24378cca9db43270686e309b63ea2a3084fa3b059154a7c99: Status 404 returned error can't find the container with id 230e79af9b25c5d24378cca9db43270686e309b63ea2a3084fa3b059154a7c99 Jul 06 12:57:00.963863 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.963751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" event={"ID":"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b","Type":"ContainerStarted","Data":"25d1164b9b5393a12490a0b9ed2b4708a522bbca036952540ccc4577a8a70416"} Jul 06 12:57:00.965770 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.965749 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerID="ed6a57b096b0bce86a597648e55706f753ec9f848232ff6fd71ebe89fd3bdbf8" exitCode=0 Jul 06 12:57:00.965864 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:00.965828 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" event={"ID":"dcdb7470-71d1-4292-b587-a09ff2e027fb","Type":"ContainerDied","Data":"ed6a57b096b0bce86a597648e55706f753ec9f848232ff6fd71ebe89fd3bdbf8"} Jul 06 12:57:01.972247 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:01.972211 2568 generic.go:356] "Generic (PLEG): container finished" podID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerID="25d1164b9b5393a12490a0b9ed2b4708a522bbca036952540ccc4577a8a70416" exitCode=0 Jul 06 12:57:01.972620 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:01.972297 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" event={"ID":"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b","Type":"ContainerDied","Data":"25d1164b9b5393a12490a0b9ed2b4708a522bbca036952540ccc4577a8a70416"} Jul 06 12:57:01.974523 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:01.974503 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerID="9cbf2295a71d550a1dbbaba62f2547f3761d0dc034f35cc1c2a8341ca2046557" exitCode=0 Jul 06 12:57:01.974604 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:01.974579 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" event={"ID":"dcdb7470-71d1-4292-b587-a09ff2e027fb","Type":"ContainerDied","Data":"9cbf2295a71d550a1dbbaba62f2547f3761d0dc034f35cc1c2a8341ca2046557"} Jul 06 12:57:01.979052 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:01.979032 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9100767-6c78-4bea-8152-c15731093106" containerID="0d4c40d0c00110ece1a171ee828c6358806c34074e2acaf251b46c5853f8ee46" exitCode=0 Jul 06 12:57:01.979151 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:01.979102 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" event={"ID":"f9100767-6c78-4bea-8152-c15731093106","Type":"ContainerDied","Data":"0d4c40d0c00110ece1a171ee828c6358806c34074e2acaf251b46c5853f8ee46"} Jul 06 12:57:01.979151 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:01.979130 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" event={"ID":"f9100767-6c78-4bea-8152-c15731093106","Type":"ContainerStarted","Data":"230e79af9b25c5d24378cca9db43270686e309b63ea2a3084fa3b059154a7c99"} Jul 06 12:57:02.985821 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:02.985791 2568 generic.go:356] "Generic (PLEG): container finished" podID="9e665150-004d-4d22-a98a-40ffeb477e21" containerID="9faaa6eea2e45e4604f8ecc93bcf95a5f71cc12de6f195255930571fcc0ce41e" exitCode=0 Jul 06 12:57:02.986295 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:02.985879 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" event={"ID":"9e665150-004d-4d22-a98a-40ffeb477e21","Type":"ContainerDied","Data":"9faaa6eea2e45e4604f8ecc93bcf95a5f71cc12de6f195255930571fcc0ce41e"} Jul 06 12:57:02.988068 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:02.988050 2568 generic.go:356] "Generic (PLEG): container finished" podID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerID="3af894a77883e0969851c1078d218089ae9912955784a63c1cfa27e30f85ebd5" exitCode=0 Jul 06 12:57:02.988167 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:02.988076 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" event={"ID":"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b","Type":"ContainerDied","Data":"3af894a77883e0969851c1078d218089ae9912955784a63c1cfa27e30f85ebd5"} Jul 06 12:57:03.216768 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.216745 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:57:03.279836 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.279811 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-util\") pod \"dcdb7470-71d1-4292-b587-a09ff2e027fb\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " Jul 06 12:57:03.279961 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.279861 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-bundle\") pod \"dcdb7470-71d1-4292-b587-a09ff2e027fb\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " Jul 06 12:57:03.279961 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.279921 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j67f\" (UniqueName: \"kubernetes.io/projected/dcdb7470-71d1-4292-b587-a09ff2e027fb-kube-api-access-5j67f\") pod \"dcdb7470-71d1-4292-b587-a09ff2e027fb\" (UID: \"dcdb7470-71d1-4292-b587-a09ff2e027fb\") " Jul 06 12:57:03.280384 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.280353 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-bundle" (OuterVolumeSpecName: "bundle") pod "dcdb7470-71d1-4292-b587-a09ff2e027fb" (UID: "dcdb7470-71d1-4292-b587-a09ff2e027fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:03.282288 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.282258 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdb7470-71d1-4292-b587-a09ff2e027fb-kube-api-access-5j67f" (OuterVolumeSpecName: "kube-api-access-5j67f") pod "dcdb7470-71d1-4292-b587-a09ff2e027fb" (UID: "dcdb7470-71d1-4292-b587-a09ff2e027fb"). InnerVolumeSpecName "kube-api-access-5j67f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:57:03.285834 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.285812 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-util" (OuterVolumeSpecName: "util") pod "dcdb7470-71d1-4292-b587-a09ff2e027fb" (UID: "dcdb7470-71d1-4292-b587-a09ff2e027fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:03.381401 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.381369 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-util\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:03.381401 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.381399 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcdb7470-71d1-4292-b587-a09ff2e027fb-bundle\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:03.381401 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.381409 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j67f\" (UniqueName: \"kubernetes.io/projected/dcdb7470-71d1-4292-b587-a09ff2e027fb-kube-api-access-5j67f\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:03.994512 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.994484 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" Jul 06 12:57:03.994512 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.994496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69" event={"ID":"dcdb7470-71d1-4292-b587-a09ff2e027fb","Type":"ContainerDied","Data":"dfa6780fefc510adc2b1db5f9bf7def12f2adc77519b006b84e0aea929c1215f"} Jul 06 12:57:03.994998 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.994550 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa6780fefc510adc2b1db5f9bf7def12f2adc77519b006b84e0aea929c1215f" Jul 06 12:57:03.996321 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.996296 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9100767-6c78-4bea-8152-c15731093106" containerID="ea8ed40d9898bde03a1a5729dbe35edc335e9fb0cafe661308cb9d2d74693285" exitCode=0 Jul 06 12:57:03.996465 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.996360 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" event={"ID":"f9100767-6c78-4bea-8152-c15731093106","Type":"ContainerDied","Data":"ea8ed40d9898bde03a1a5729dbe35edc335e9fb0cafe661308cb9d2d74693285"} Jul 06 12:57:03.998866 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.998841 2568 generic.go:356] "Generic (PLEG): container finished" podID="9e665150-004d-4d22-a98a-40ffeb477e21" containerID="1d1e563eb3efbb528928d52fd23140b1d88811e2f7631b6886bb2f7eb7fcd74a" exitCode=0 Jul 06 12:57:03.998956 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:03.998911 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" event={"ID":"9e665150-004d-4d22-a98a-40ffeb477e21","Type":"ContainerDied","Data":"1d1e563eb3efbb528928d52fd23140b1d88811e2f7631b6886bb2f7eb7fcd74a"} Jul 06 12:57:04.141201 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.141161 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:57:04.289994 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.289896 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6tb8\" (UniqueName: \"kubernetes.io/projected/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-kube-api-access-n6tb8\") pod \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " Jul 06 12:57:04.289994 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.289946 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-bundle\") pod \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " Jul 06 12:57:04.290210 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.290023 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-util\") pod \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\" (UID: \"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b\") " Jul 06 12:57:04.290693 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.290659 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-bundle" (OuterVolumeSpecName: "bundle") pod "b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" (UID: "b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:04.292538 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.292507 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-kube-api-access-n6tb8" (OuterVolumeSpecName: "kube-api-access-n6tb8") pod "b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" (UID: "b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b"). InnerVolumeSpecName "kube-api-access-n6tb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:57:04.296557 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.296523 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-util" (OuterVolumeSpecName: "util") pod "b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" (UID: "b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:04.390684 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.390616 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6tb8\" (UniqueName: \"kubernetes.io/projected/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-kube-api-access-n6tb8\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:04.390684 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.390679 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-bundle\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:04.390888 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:04.390696 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b-util\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:05.009828 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.009801 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" Jul 06 12:57:05.009828 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.009816 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6" event={"ID":"b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b","Type":"ContainerDied","Data":"04d663151a6bc298ba6b5b7f39e5d81974601bd8053f4eb033fb3702e02c5a6e"} Jul 06 12:57:05.010238 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.009860 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d663151a6bc298ba6b5b7f39e5d81974601bd8053f4eb033fb3702e02c5a6e" Jul 06 12:57:05.012168 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.012143 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9100767-6c78-4bea-8152-c15731093106" containerID="edd6098cf02b548b52572c0b8bbbe98715406b1b7c4b69d594b93d44bc109be3" exitCode=0 Jul 06 12:57:05.012276 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.012238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" event={"ID":"f9100767-6c78-4bea-8152-c15731093106","Type":"ContainerDied","Data":"edd6098cf02b548b52572c0b8bbbe98715406b1b7c4b69d594b93d44bc109be3"} Jul 06 12:57:05.141876 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.141845 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:05.298887 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.298785 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4kqp\" (UniqueName: \"kubernetes.io/projected/9e665150-004d-4d22-a98a-40ffeb477e21-kube-api-access-c4kqp\") pod \"9e665150-004d-4d22-a98a-40ffeb477e21\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " Jul 06 12:57:05.298887 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.298880 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-util\") pod \"9e665150-004d-4d22-a98a-40ffeb477e21\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " Jul 06 12:57:05.299115 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.298914 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-bundle\") pod \"9e665150-004d-4d22-a98a-40ffeb477e21\" (UID: \"9e665150-004d-4d22-a98a-40ffeb477e21\") " Jul 06 12:57:05.299533 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.299508 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-bundle" (OuterVolumeSpecName: "bundle") pod "9e665150-004d-4d22-a98a-40ffeb477e21" (UID: "9e665150-004d-4d22-a98a-40ffeb477e21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:05.301038 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.301014 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e665150-004d-4d22-a98a-40ffeb477e21-kube-api-access-c4kqp" (OuterVolumeSpecName: "kube-api-access-c4kqp") pod "9e665150-004d-4d22-a98a-40ffeb477e21" (UID: "9e665150-004d-4d22-a98a-40ffeb477e21"). InnerVolumeSpecName "kube-api-access-c4kqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:57:05.304349 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.304329 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-util" (OuterVolumeSpecName: "util") pod "9e665150-004d-4d22-a98a-40ffeb477e21" (UID: "9e665150-004d-4d22-a98a-40ffeb477e21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:05.400399 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.400362 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4kqp\" (UniqueName: \"kubernetes.io/projected/9e665150-004d-4d22-a98a-40ffeb477e21-kube-api-access-c4kqp\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:05.400399 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.400395 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-util\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:05.400399 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:05.400404 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e665150-004d-4d22-a98a-40ffeb477e21-bundle\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:06.019230 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.019202 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" Jul 06 12:57:06.019583 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.019230 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m" event={"ID":"9e665150-004d-4d22-a98a-40ffeb477e21","Type":"ContainerDied","Data":"e90144f04fe72580c44203e734f40027d5ef75b0ef807ae7608539e142080959"} Jul 06 12:57:06.019583 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.019275 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e90144f04fe72580c44203e734f40027d5ef75b0ef807ae7608539e142080959" Jul 06 12:57:06.151335 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.151309 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:06.308856 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.308771 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-util\") pod \"f9100767-6c78-4bea-8152-c15731093106\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " Jul 06 12:57:06.308856 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.308821 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9qtk\" (UniqueName: \"kubernetes.io/projected/f9100767-6c78-4bea-8152-c15731093106-kube-api-access-c9qtk\") pod \"f9100767-6c78-4bea-8152-c15731093106\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " Jul 06 12:57:06.309185 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.308894 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-bundle\") pod \"f9100767-6c78-4bea-8152-c15731093106\" (UID: \"f9100767-6c78-4bea-8152-c15731093106\") " Jul 06 12:57:06.309433 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.309402 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-bundle" (OuterVolumeSpecName: "bundle") pod "f9100767-6c78-4bea-8152-c15731093106" (UID: "f9100767-6c78-4bea-8152-c15731093106"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:06.311141 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.311117 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9100767-6c78-4bea-8152-c15731093106-kube-api-access-c9qtk" (OuterVolumeSpecName: "kube-api-access-c9qtk") pod "f9100767-6c78-4bea-8152-c15731093106" (UID: "f9100767-6c78-4bea-8152-c15731093106"). InnerVolumeSpecName "kube-api-access-c9qtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:57:06.314260 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.314238 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-util" (OuterVolumeSpecName: "util") pod "f9100767-6c78-4bea-8152-c15731093106" (UID: "f9100767-6c78-4bea-8152-c15731093106"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:06.410376 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.410343 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-bundle\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:06.410376 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.410371 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9100767-6c78-4bea-8152-c15731093106-util\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:06.410376 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:06.410381 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9qtk\" (UniqueName: \"kubernetes.io/projected/f9100767-6c78-4bea-8152-c15731093106-kube-api-access-c9qtk\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:07.025362 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:07.025331 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" Jul 06 12:57:07.025806 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:07.025357 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2" event={"ID":"f9100767-6c78-4bea-8152-c15731093106","Type":"ContainerDied","Data":"230e79af9b25c5d24378cca9db43270686e309b63ea2a3084fa3b059154a7c99"} Jul 06 12:57:07.025806 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:07.025447 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="230e79af9b25c5d24378cca9db43270686e309b63ea2a3084fa3b059154a7c99" Jul 06 12:57:15.378343 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378302 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x"] Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378741 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerName="util" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378759 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerName="util" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378774 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerName="extract" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378782 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerName="extract" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378791 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerName="util" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378799 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerName="util" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378808 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerName="pull" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378815 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerName="pull" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378835 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e665150-004d-4d22-a98a-40ffeb477e21" containerName="util" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378843 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e665150-004d-4d22-a98a-40ffeb477e21" containerName="util" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378852 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e665150-004d-4d22-a98a-40ffeb477e21" containerName="pull" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378859 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e665150-004d-4d22-a98a-40ffeb477e21" containerName="pull" Jul 06 12:57:15.378854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378869 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e665150-004d-4d22-a98a-40ffeb477e21" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378877 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e665150-004d-4d22-a98a-40ffeb477e21" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378886 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9100767-6c78-4bea-8152-c15731093106" containerName="pull" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378893 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9100767-6c78-4bea-8152-c15731093106" containerName="pull" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378909 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378917 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378928 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9100767-6c78-4bea-8152-c15731093106" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378936 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9100767-6c78-4bea-8152-c15731093106" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378944 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerName="pull" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378952 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerName="pull" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378969 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9100767-6c78-4bea-8152-c15731093106" containerName="util" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.378979 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9100767-6c78-4bea-8152-c15731093106" containerName="util" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.379059 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcdb7470-71d1-4292-b587-a09ff2e027fb" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.379072 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9100767-6c78-4bea-8152-c15731093106" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.379087 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b" containerName="extract" Jul 06 12:57:15.379480 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.379095 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e665150-004d-4d22-a98a-40ffeb477e21" containerName="extract" Jul 06 12:57:15.382604 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.382585 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" Jul 06 12:57:15.385209 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.385188 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 12:57:15.385307 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.385267 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zzf78\"" Jul 06 12:57:15.395515 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.395495 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x"] Jul 06 12:57:15.495238 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.495204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmbs\" (UniqueName: \"kubernetes.io/projected/81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f-kube-api-access-nsmbs\") pod \"dns-operator-controller-manager-864dd78c4c-gb27x\" (UID: \"81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" Jul 06 12:57:15.595972 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.595939 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmbs\" (UniqueName: \"kubernetes.io/projected/81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f-kube-api-access-nsmbs\") pod \"dns-operator-controller-manager-864dd78c4c-gb27x\" (UID: \"81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" Jul 06 12:57:15.608223 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.608197 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmbs\" (UniqueName: \"kubernetes.io/projected/81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f-kube-api-access-nsmbs\") pod \"dns-operator-controller-manager-864dd78c4c-gb27x\" (UID: \"81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" Jul 06 12:57:15.692720 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.692624 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" Jul 06 12:57:15.850192 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:15.850165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x"] Jul 06 12:57:15.852253 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:15.852223 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d84d1f_8a68_4bd2_86dc_b9f9236a0d4f.slice/crio-320d655560d6c3b5e074ce71e783cb78f6ab871155de295ad842c696a30a1037 WatchSource:0}: Error finding container 320d655560d6c3b5e074ce71e783cb78f6ab871155de295ad842c696a30a1037: Status 404 returned error can't find the container with id 320d655560d6c3b5e074ce71e783cb78f6ab871155de295ad842c696a30a1037 Jul 06 12:57:16.068448 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:16.068372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" event={"ID":"81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f","Type":"ContainerStarted","Data":"320d655560d6c3b5e074ce71e783cb78f6ab871155de295ad842c696a30a1037"} Jul 06 12:57:18.953133 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:18.953097 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4"] Jul 06 12:57:18.956483 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:18.956466 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:18.960528 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:18.960507 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x4fw7\"" Jul 06 12:57:18.978073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:18.978046 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4"] Jul 06 12:57:19.022687 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.022660 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8lp\" (UniqueName: \"kubernetes.io/projected/1dcb6757-49c2-4f83-8ec5-f02ae6b2c064-kube-api-access-td8lp\") pod \"limitador-operator-controller-manager-859d7b78bd-7prq4\" (UID: \"1dcb6757-49c2-4f83-8ec5-f02ae6b2c064\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:19.085389 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.085363 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" event={"ID":"81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f","Type":"ContainerStarted","Data":"dcfb7eaa1d252be8fc617fdbfdff52d76e0565b06f0784fc3828f093386f038c"} Jul 06 12:57:19.085557 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.085427 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" Jul 06 12:57:19.123132 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.123100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8lp\" (UniqueName: \"kubernetes.io/projected/1dcb6757-49c2-4f83-8ec5-f02ae6b2c064-kube-api-access-td8lp\") pod \"limitador-operator-controller-manager-859d7b78bd-7prq4\" (UID: \"1dcb6757-49c2-4f83-8ec5-f02ae6b2c064\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:19.137033 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.136992 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" podStartSLOduration=1.7985142600000001 podStartE2EDuration="4.136979355s" totalImagesPullingTime="2.338465095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:57:19.131935274 +0000 UTC m=+623.072776132" watchObservedRunningTime="2026-07-06 12:57:19.136979355 +0000 UTC m=+623.077820213" Jul 06 12:57:19.162728 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.162703 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8lp\" (UniqueName: \"kubernetes.io/projected/1dcb6757-49c2-4f83-8ec5-f02ae6b2c064-kube-api-access-td8lp\") pod \"limitador-operator-controller-manager-859d7b78bd-7prq4\" (UID: \"1dcb6757-49c2-4f83-8ec5-f02ae6b2c064\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:19.266945 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.266868 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:19.436242 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:19.436215 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4"] Jul 06 12:57:19.437541 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:19.437506 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcb6757_49c2_4f83_8ec5_f02ae6b2c064.slice/crio-6c08be2f647da3701f1d8fea5cd8990e6c553a3292ca667b72a6df07c3b199f1 WatchSource:0}: Error finding container 6c08be2f647da3701f1d8fea5cd8990e6c553a3292ca667b72a6df07c3b199f1: Status 404 returned error can't find the container with id 6c08be2f647da3701f1d8fea5cd8990e6c553a3292ca667b72a6df07c3b199f1 Jul 06 12:57:20.092696 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:20.092662 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" event={"ID":"1dcb6757-49c2-4f83-8ec5-f02ae6b2c064","Type":"ContainerStarted","Data":"6c08be2f647da3701f1d8fea5cd8990e6c553a3292ca667b72a6df07c3b199f1"} Jul 06 12:57:22.106049 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:22.106004 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" event={"ID":"1dcb6757-49c2-4f83-8ec5-f02ae6b2c064","Type":"ContainerStarted","Data":"cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02"} Jul 06 12:57:22.106492 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:22.106139 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:24.212841 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.207708 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" podStartSLOduration=4.610969405 podStartE2EDuration="6.207687274s" totalImagesPullingTime="1.596717869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:57:22.133939827 +0000 UTC m=+626.074780686" watchObservedRunningTime="2026-07-06 12:57:24.207687274 +0000 UTC m=+628.148528134" Jul 06 12:57:24.212841 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.208165 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg"] Jul 06 12:57:24.218581 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.218560 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.257165 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.257142 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6mvwm\"" Jul 06 12:57:24.268643 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.268610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.268756 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.268682 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzpd\" (UniqueName: \"kubernetes.io/projected/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-kube-api-access-4dzpd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.269107 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.269086 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg"] Jul 06 12:57:24.370107 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.370075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.370276 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.370118 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzpd\" (UniqueName: \"kubernetes.io/projected/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-kube-api-access-4dzpd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.370476 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.370446 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.391534 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.391511 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzpd\" (UniqueName: \"kubernetes.io/projected/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-kube-api-access-4dzpd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.529280 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.529210 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:24.764454 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:24.764420 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg"] Jul 06 12:57:24.764889 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:24.764860 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb67cc17_fc54_4e2a_bc64_ca988a64a4ad.slice/crio-08ccce6d49252bd705525068d51e6e30ecff603c4cd85b6cb6924f4ee9bede08 WatchSource:0}: Error finding container 08ccce6d49252bd705525068d51e6e30ecff603c4cd85b6cb6924f4ee9bede08: Status 404 returned error can't find the container with id 08ccce6d49252bd705525068d51e6e30ecff603c4cd85b6cb6924f4ee9bede08 Jul 06 12:57:25.122852 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:25.122820 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" event={"ID":"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad","Type":"ContainerStarted","Data":"08ccce6d49252bd705525068d51e6e30ecff603c4cd85b6cb6924f4ee9bede08"} Jul 06 12:57:30.094912 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:30.094885 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gb27x" Jul 06 12:57:30.155932 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:30.155892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" event={"ID":"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad","Type":"ContainerStarted","Data":"b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5"} Jul 06 12:57:30.159790 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:30.159762 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:30.231063 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:30.230765 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" podStartSLOduration=1.711055863 podStartE2EDuration="6.230745935s" totalImagesPullingTime="4.519690072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:57:30.227600619 +0000 UTC m=+634.168441489" watchObservedRunningTime="2026-07-06 12:57:30.230745935 +0000 UTC m=+634.171586798" Jul 06 12:57:32.373536 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.373503 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s"] Jul 06 12:57:32.376978 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.376959 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.379750 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.379729 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 12:57:32.379854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.379729 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 12:57:32.379854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.379824 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7v8gc\"" Jul 06 12:57:32.400870 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.400849 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s"] Jul 06 12:57:32.441467 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.441441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.441561 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.441503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82hg\" (UniqueName: \"kubernetes.io/projected/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-kube-api-access-j82hg\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.441561 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.441532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.542966 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.542941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82hg\" (UniqueName: \"kubernetes.io/projected/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-kube-api-access-j82hg\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.543106 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.542974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.543106 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.542999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.543180 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:32.543099 2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 12:57:32.543180 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:32.543159 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-plugin-serving-cert podName:0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd nodeName:}" failed. No retries permitted until 2026-07-06 12:57:33.043142135 +0000 UTC m=+636.983982975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-vb42s" (UID: "0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd") : secret "plugin-serving-cert" not found Jul 06 12:57:32.543674 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.543658 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:32.558995 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:32.558970 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82hg\" (UniqueName: \"kubernetes.io/projected/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-kube-api-access-j82hg\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:33.046422 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:33.046380 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:33.048922 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:33.048893 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vb42s\" (UID: \"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:33.114707 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:33.114684 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:33.286189 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:33.286146 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" Jul 06 12:57:33.420066 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:33.420043 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s"] Jul 06 12:57:33.422051 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:33.422026 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f7f0f78_9f0c_47ea_8ee5_cea7d1ad8efd.slice/crio-28a3d278abf568d0c5414797f52ff1f89ab3f46d5e4d6a7e44caba995368973f WatchSource:0}: Error finding container 28a3d278abf568d0c5414797f52ff1f89ab3f46d5e4d6a7e44caba995368973f: Status 404 returned error can't find the container with id 28a3d278abf568d0c5414797f52ff1f89ab3f46d5e4d6a7e44caba995368973f Jul 06 12:57:34.177876 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:34.177845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" event={"ID":"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd","Type":"ContainerStarted","Data":"28a3d278abf568d0c5414797f52ff1f89ab3f46d5e4d6a7e44caba995368973f"} Jul 06 12:57:42.169886 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:42.169854 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:44.377199 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.377168 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg"] Jul 06 12:57:44.377685 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.377404 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" containerName="manager" containerID="cri-o://b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5" gracePeriod=2 Jul 06 12:57:44.383598 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.383571 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg"] Jul 06 12:57:44.411331 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.411297 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4"] Jul 06 12:57:44.411576 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.411532 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" containerName="manager" containerID="cri-o://cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02" gracePeriod=2 Jul 06 12:57:44.428432 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:44.428393 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:44.428886 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.428824 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4"] Jul 06 12:57:44.432915 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:44.432880 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7prq4\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:44.473994 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.473967 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf"] Jul 06 12:57:44.474355 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.474336 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" containerName="manager" Jul 06 12:57:44.474355 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.474357 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" containerName="manager" Jul 06 12:57:44.474510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.474380 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" containerName="manager" Jul 06 12:57:44.474510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.474389 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" containerName="manager" Jul 06 12:57:44.474510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.474462 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" containerName="manager" Jul 06 12:57:44.474510 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.474476 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" containerName="manager" Jul 06 12:57:44.479521 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.479500 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" Jul 06 12:57:44.481966 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:44.481936 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7prq4\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:44.518017 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.517984 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf"] Jul 06 12:57:44.547790 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.547757 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9h6\" (UniqueName: \"kubernetes.io/projected/8e086120-79aa-44ef-880c-936bded62191-kube-api-access-cg9h6\") pod \"limitador-operator-controller-manager-859d7b78bd-7b9lf\" (UID: \"8e086120-79aa-44ef-880c-936bded62191\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" Jul 06 12:57:44.554988 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:44.554958 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:44.648330 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.648252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9h6\" (UniqueName: \"kubernetes.io/projected/8e086120-79aa-44ef-880c-936bded62191-kube-api-access-cg9h6\") pod \"limitador-operator-controller-manager-859d7b78bd-7b9lf\" (UID: \"8e086120-79aa-44ef-880c-936bded62191\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" Jul 06 12:57:44.668264 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.668229 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9h6\" (UniqueName: \"kubernetes.io/projected/8e086120-79aa-44ef-880c-936bded62191-kube-api-access-cg9h6\") pod \"limitador-operator-controller-manager-859d7b78bd-7b9lf\" (UID: \"8e086120-79aa-44ef-880c-936bded62191\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" Jul 06 12:57:44.804519 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:44.804487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" Jul 06 12:57:46.978813 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:46.978784 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm"] Jul 06 12:57:47.057226 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.057197 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm"] Jul 06 12:57:47.057388 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.057331 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:47.172539 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.172505 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7z2\" (UniqueName: \"kubernetes.io/projected/e3c1ef71-6fa0-4080-8e89-68a3c030f686-kube-api-access-kf7z2\") pod \"kuadrant-operator-controller-manager-86878c769f-722qm\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:47.172722 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.172678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3c1ef71-6fa0-4080-8e89-68a3c030f686-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-722qm\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:47.274084 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.274000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7z2\" (UniqueName: \"kubernetes.io/projected/e3c1ef71-6fa0-4080-8e89-68a3c030f686-kube-api-access-kf7z2\") pod \"kuadrant-operator-controller-manager-86878c769f-722qm\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:47.274261 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.274131 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3c1ef71-6fa0-4080-8e89-68a3c030f686-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-722qm\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:47.274506 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.274484 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3c1ef71-6fa0-4080-8e89-68a3c030f686-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-722qm\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:47.309116 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.309088 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7z2\" (UniqueName: \"kubernetes.io/projected/e3c1ef71-6fa0-4080-8e89-68a3c030f686-kube-api-access-kf7z2\") pod \"kuadrant-operator-controller-manager-86878c769f-722qm\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:47.369047 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:47.369013 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:55.519558 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.519536 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf"] Jul 06 12:57:55.524945 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:55.524923 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e086120_79aa_44ef_880c_936bded62191.slice/crio-ec9ccce46891619799a85d9b964ffb2e2b20d1ec50352eecd5a7c9351a00195f WatchSource:0}: Error finding container ec9ccce46891619799a85d9b964ffb2e2b20d1ec50352eecd5a7c9351a00195f: Status 404 returned error can't find the container with id ec9ccce46891619799a85d9b964ffb2e2b20d1ec50352eecd5a7c9351a00195f Jul 06 12:57:55.531504 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.531192 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:55.533324 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:55.533208 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:55.534853 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.534832 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:55.536692 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:55.536600 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:55.538384 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:55.538353 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7prq4\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:55.549759 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.549712 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm"] Jul 06 12:57:55.561899 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:57:55.561877 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c1ef71_6fa0_4080_8e89_68a3c030f686.slice/crio-d6e9bfff3422e60947ffe24e420792d51a01f8fab5e0cf14f4c7e02b3f4ffa90 WatchSource:0}: Error finding container d6e9bfff3422e60947ffe24e420792d51a01f8fab5e0cf14f4c7e02b3f4ffa90: Status 404 returned error can't find the container with id d6e9bfff3422e60947ffe24e420792d51a01f8fab5e0cf14f4c7e02b3f4ffa90 Jul 06 12:57:55.655582 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.655554 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-extensions-socket-volume\") pod \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " Jul 06 12:57:55.655718 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.655617 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td8lp\" (UniqueName: \"kubernetes.io/projected/1dcb6757-49c2-4f83-8ec5-f02ae6b2c064-kube-api-access-td8lp\") pod \"1dcb6757-49c2-4f83-8ec5-f02ae6b2c064\" (UID: \"1dcb6757-49c2-4f83-8ec5-f02ae6b2c064\") " Jul 06 12:57:55.655774 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.655718 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dzpd\" (UniqueName: \"kubernetes.io/projected/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-kube-api-access-4dzpd\") pod \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\" (UID: \"fb67cc17-fc54-4e2a-bc64-ca988a64a4ad\") " Jul 06 12:57:55.656184 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.656157 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" (UID: "fb67cc17-fc54-4e2a-bc64-ca988a64a4ad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:57:55.657933 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.657911 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-kube-api-access-4dzpd" (OuterVolumeSpecName: "kube-api-access-4dzpd") pod "fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" (UID: "fb67cc17-fc54-4e2a-bc64-ca988a64a4ad"). InnerVolumeSpecName "kube-api-access-4dzpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:57:55.658031 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.657965 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcb6757-49c2-4f83-8ec5-f02ae6b2c064-kube-api-access-td8lp" (OuterVolumeSpecName: "kube-api-access-td8lp") pod "1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" (UID: "1dcb6757-49c2-4f83-8ec5-f02ae6b2c064"). InnerVolumeSpecName "kube-api-access-td8lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:57:55.757388 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.757350 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dzpd\" (UniqueName: \"kubernetes.io/projected/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-kube-api-access-4dzpd\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:55.757388 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.757388 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad-extensions-socket-volume\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:55.757586 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:55.757405 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td8lp\" (UniqueName: \"kubernetes.io/projected/1dcb6757-49c2-4f83-8ec5-f02ae6b2c064-kube-api-access-td8lp\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 12:57:56.305090 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.305053 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" event={"ID":"0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd","Type":"ContainerStarted","Data":"d747aae3528f114f9e640dd58e46afc1ed596bae485bfc49d57f88ab90c8188a"} Jul 06 12:57:56.306334 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.306308 2568 generic.go:356] "Generic (PLEG): container finished" podID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" containerID="cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02" exitCode=0 Jul 06 12:57:56.306450 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.306339 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:56.306450 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.306383 2568 scope.go:117] "RemoveContainer" containerID="cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02" Jul 06 12:57:56.307402 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.307370 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:56.308088 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.308063 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" event={"ID":"e3c1ef71-6fa0-4080-8e89-68a3c030f686","Type":"ContainerStarted","Data":"c2d8f8680959f5ae1eeffdb289ca95fe01c677727d8756c5201b34a8c1ff8e3e"} Jul 06 12:57:56.308193 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.308115 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" event={"ID":"e3c1ef71-6fa0-4080-8e89-68a3c030f686","Type":"ContainerStarted","Data":"d6e9bfff3422e60947ffe24e420792d51a01f8fab5e0cf14f4c7e02b3f4ffa90"} Jul 06 12:57:56.308193 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.308146 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:57:56.309425 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.309400 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7prq4\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:56.309888 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.309866 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" event={"ID":"8e086120-79aa-44ef-880c-936bded62191","Type":"ContainerStarted","Data":"a416ce4282b1e69cd83e17350f50b9b60a0d0066044c73d97f6ce860eef7fb5a"} Jul 06 12:57:56.309977 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.309907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" event={"ID":"8e086120-79aa-44ef-880c-936bded62191","Type":"ContainerStarted","Data":"ec9ccce46891619799a85d9b964ffb2e2b20d1ec50352eecd5a7c9351a00195f"} Jul 06 12:57:56.309977 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.309951 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" Jul 06 12:57:56.311296 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.311274 2568 generic.go:356] "Generic (PLEG): container finished" podID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" containerID="b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5" exitCode=0 Jul 06 12:57:56.311392 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.311321 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:56.315992 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.315974 2568 scope.go:117] "RemoveContainer" containerID="cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02" Jul 06 12:57:56.316248 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.316231 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02\": container with ID starting with cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02 not found: ID does not exist" containerID="cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02" Jul 06 12:57:56.316304 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.316256 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02"} err="failed to get container status \"cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02\": rpc error: code = NotFound desc = could not find container \"cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02\": container with ID starting with cd77c545e798695e0f1e4b5d5199d2b6801e885d37f4175e8e18f37756979f02 not found: ID does not exist" Jul 06 12:57:56.316304 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.316281 2568 scope.go:117] "RemoveContainer" containerID="b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5" Jul 06 12:57:56.333227 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.333181 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vb42s" podStartSLOduration=2.334310264 podStartE2EDuration="24.333164245s" totalImagesPullingTime="21.998853981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:57:56.332673224 +0000 UTC m=+660.273514087" watchObservedRunningTime="2026-07-06 12:57:56.333164245 +0000 UTC m=+660.274005116" Jul 06 12:57:56.337073 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.336900 2568 scope.go:117] "RemoveContainer" containerID="b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5" Jul 06 12:57:56.337168 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.337150 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5\": container with ID starting with b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5 not found: ID does not exist" containerID="b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5" Jul 06 12:57:56.337212 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.337175 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5"} err="failed to get container status \"b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5\": rpc error: code = NotFound desc = could not find container \"b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5\": container with ID starting with b9b12d73c8ff92d67b4be284021db772defee48a7576182906d399c73e9ca2e5 not found: ID does not exist" Jul 06 12:57:56.359148 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.359100 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" podStartSLOduration=10.359086229 podStartE2EDuration="10.359086229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:57:56.357564578 +0000 UTC m=+660.298405451" watchObservedRunningTime="2026-07-06 12:57:56.359086229 +0000 UTC m=+660.299927087" Jul 06 12:57:56.359602 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.359564 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:56.361467 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.361435 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7prq4\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:56.380471 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.380213 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" podStartSLOduration=12.380194134 podStartE2EDuration="12.380194134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:57:56.380037303 +0000 UTC m=+660.320878175" watchObservedRunningTime="2026-07-06 12:57:56.380194134 +0000 UTC m=+660.321034994" Jul 06 12:57:56.382065 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.382041 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:56.383926 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.383899 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7prq4\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:57:56.590675 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.590584 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7k5bg\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7k5bg" Jul 06 12:57:56.591747 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.591720 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" path="/var/lib/kubelet/pods/1dcb6757-49c2-4f83-8ec5-f02ae6b2c064/volumes" Jul 06 12:57:56.592152 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:57:56.592133 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb67cc17-fc54-4e2a-bc64-ca988a64a4ad" path="/var/lib/kubelet/pods/fb67cc17-fc54-4e2a-bc64-ca988a64a4ad/volumes" Jul 06 12:57:56.592610 ip-10-0-138-145 kubenswrapper[2568]: E0706 12:57:56.592588 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7prq4\" is forbidden: User \"system:node:ip-10-0-138-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-145.ec2.internal' and this object" podUID="1dcb6757-49c2-4f83-8ec5-f02ae6b2c064" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7prq4" Jul 06 12:58:07.322134 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:07.322104 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b9lf" Jul 06 12:58:07.322668 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:07.322646 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 12:58:23.848884 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.848849 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c"] Jul 06 12:58:23.867413 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.867386 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.867854 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.867828 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c"] Jul 06 12:58:23.869984 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.869964 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ws7pj\"" Jul 06 12:58:23.977590 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977559 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-workload-socket\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977781 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977596 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-token\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977781 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjnn\" (UniqueName: \"kubernetes.io/projected/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-kube-api-access-4jjnn\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977781 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-credential-socket\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977781 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977712 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977920 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977920 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977920 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-workload-certs\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:23.977920 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:23.977883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-data\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.078618 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjnn\" (UniqueName: \"kubernetes.io/projected/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-kube-api-access-4jjnn\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.078618 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-credential-socket\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.078855 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078668 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.078855 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.078855 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.078855 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-workload-certs\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.078855 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078832 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-data\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.079104 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078861 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-workload-socket\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.079104 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.078899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-token\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.079104 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.079061 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-credential-socket\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.079247 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.079214 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-workload-certs\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.079317 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.079293 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-workload-socket\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.079379 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.079363 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-data\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.079462 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.079444 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.081238 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.081218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.081485 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.081465 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.090201 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.090171 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjnn\" (UniqueName: \"kubernetes.io/projected/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-kube-api-access-4jjnn\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.090286 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.090245 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a163c8-f10d-4f9d-8c64-e1b1fd92a185-istio-token\") pod \"maas-default-gateway-openshift-default-5d7b8c9489-rn98c\" (UID: \"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.180127 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.180048 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:24.329668 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.329625 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c"] Jul 06 12:58:24.331027 ip-10-0-138-145 kubenswrapper[2568]: W0706 12:58:24.331001 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a163c8_f10d_4f9d_8c64_e1b1fd92a185.slice/crio-d62b6d6bc01423a133d6781b7bd53d65527b6afac6a2050f064d1637ee674eab WatchSource:0}: Error finding container d62b6d6bc01423a133d6781b7bd53d65527b6afac6a2050f064d1637ee674eab: Status 404 returned error can't find the container with id d62b6d6bc01423a133d6781b7bd53d65527b6afac6a2050f064d1637ee674eab Jul 06 12:58:24.333011 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.332980 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:58:24.333068 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.333046 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:58:24.333104 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.333074 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:58:24.459127 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.459043 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" event={"ID":"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185","Type":"ContainerStarted","Data":"85b2a71a69f5ca399182678d88bdf28d66f4fb02524eaf63c038ce2006832a4f"} Jul 06 12:58:24.459127 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.459104 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" event={"ID":"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185","Type":"ContainerStarted","Data":"d62b6d6bc01423a133d6781b7bd53d65527b6afac6a2050f064d1637ee674eab"} Jul 06 12:58:24.481693 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:24.481647 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" podStartSLOduration=1.481613778 podStartE2EDuration="1.481613778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:58:24.48047465 +0000 UTC m=+688.421315507" watchObservedRunningTime="2026-07-06 12:58:24.481613778 +0000 UTC m=+688.422454636" Jul 06 12:58:25.180380 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:25.180348 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:25.185282 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:25.185258 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:25.470711 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:25.470616 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:58:25.471574 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:58:25.471557 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:59:53.918434 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:53.918401 2568 generic.go:356] "Generic (PLEG): container finished" podID="e0a163c8-f10d-4f9d-8c64-e1b1fd92a185" containerID="85b2a71a69f5ca399182678d88bdf28d66f4fb02524eaf63c038ce2006832a4f" exitCode=0 Jul 06 12:59:53.918975 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:53.918469 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" event={"ID":"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185","Type":"ContainerDied","Data":"85b2a71a69f5ca399182678d88bdf28d66f4fb02524eaf63c038ce2006832a4f"} Jul 06 12:59:53.918975 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:53.918939 2568 scope.go:117] "RemoveContainer" containerID="85b2a71a69f5ca399182678d88bdf28d66f4fb02524eaf63c038ce2006832a4f" Jul 06 12:59:53.919985 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:53.919956 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:59:53.920061 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:53.920041 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:59:53.920104 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:53.920089 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 12:59:54.924434 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:54.924402 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" event={"ID":"e0a163c8-f10d-4f9d-8c64-e1b1fd92a185","Type":"ContainerStarted","Data":"68d91a22baf28692908f0793ac06459313dd50a43994dcb62e80f22d0d64f690"} Jul 06 12:59:55.180646 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:55.180534 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:59:55.181766 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:55.181742 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5d7b8c9489-rn98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 06 12:59:55.181829 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:55.181806 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" podUID="e0a163c8-f10d-4f9d-8c64-e1b1fd92a185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 06 12:59:55.471215 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:55.471061 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:59:56.180601 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:56.180552 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5d7b8c9489-rn98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 06 12:59:56.181109 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:56.180609 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" podUID="e0a163c8-f10d-4f9d-8c64-e1b1fd92a185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 06 12:59:57.181244 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:57.181200 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5d7b8c9489-rn98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 06 12:59:57.181751 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:57.181265 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" podUID="e0a163c8-f10d-4f9d-8c64-e1b1fd92a185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 06 12:59:58.181407 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:58.181369 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5d7b8c9489-rn98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 06 12:59:58.181845 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:58.181426 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" podUID="e0a163c8-f10d-4f9d-8c64-e1b1fd92a185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 06 12:59:59.184607 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:59.184578 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 12:59:59.185602 ip-10-0-138-145 kubenswrapper[2568]: I0706 12:59:59.185585 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d7b8c9489-rn98c" Jul 06 13:00:00.151451 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.151425 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722380-x64h8"] Jul 06 13:00:00.155291 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.155271 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" Jul 06 13:00:00.157786 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.157767 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-5pfrb\"" Jul 06 13:00:00.163390 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.163358 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722380-x64h8"] Jul 06 13:00:00.267656 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.267603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2qv\" (UniqueName: \"kubernetes.io/projected/da67aed7-44a3-45f2-95d7-0a44bae09e18-kube-api-access-2w2qv\") pod \"maas-api-key-cleanup-29722380-x64h8\" (UID: \"da67aed7-44a3-45f2-95d7-0a44bae09e18\") " pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" Jul 06 13:00:00.368961 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.368931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2qv\" (UniqueName: \"kubernetes.io/projected/da67aed7-44a3-45f2-95d7-0a44bae09e18-kube-api-access-2w2qv\") pod \"maas-api-key-cleanup-29722380-x64h8\" (UID: \"da67aed7-44a3-45f2-95d7-0a44bae09e18\") " pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" Jul 06 13:00:00.381012 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.380983 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2qv\" (UniqueName: \"kubernetes.io/projected/da67aed7-44a3-45f2-95d7-0a44bae09e18-kube-api-access-2w2qv\") pod \"maas-api-key-cleanup-29722380-x64h8\" (UID: \"da67aed7-44a3-45f2-95d7-0a44bae09e18\") " pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" Jul 06 13:00:00.467018 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.466939 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" Jul 06 13:00:00.602492 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.602464 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722380-x64h8"] Jul 06 13:00:00.604142 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:00:00.604116 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda67aed7_44a3_45f2_95d7_0a44bae09e18.slice/crio-dee6cb3c4a8e4fbf58013c3f9c4c7c9581d1b5fb75d5382b7cf48c80409f5a61 WatchSource:0}: Error finding container dee6cb3c4a8e4fbf58013c3f9c4c7c9581d1b5fb75d5382b7cf48c80409f5a61: Status 404 returned error can't find the container with id dee6cb3c4a8e4fbf58013c3f9c4c7c9581d1b5fb75d5382b7cf48c80409f5a61 Jul 06 13:00:00.605858 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.605839 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 13:00:00.964998 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:00.964959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" event={"ID":"da67aed7-44a3-45f2-95d7-0a44bae09e18","Type":"ContainerStarted","Data":"dee6cb3c4a8e4fbf58013c3f9c4c7c9581d1b5fb75d5382b7cf48c80409f5a61"} Jul 06 13:00:13.219990 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.219953 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55"] Jul 06 13:00:13.224922 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.224899 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.227577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.227556 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 13:00:13.228577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.228555 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 13:00:13.228698 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.228580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j6ntn\"" Jul 06 13:00:13.228698 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.228589 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 13:00:13.242867 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.242843 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55"] Jul 06 13:00:13.381284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.381246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.381284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.381279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.381534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.381308 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.381534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.381329 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.381534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.381349 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpll\" (UniqueName: \"kubernetes.io/projected/22067628-2cb6-4c99-b404-7d4ccd6231df-kube-api-access-qjpll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.381534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.381446 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22067628-2cb6-4c99-b404-7d4ccd6231df-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482126 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482126 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482126 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482094 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpll\" (UniqueName: \"kubernetes.io/projected/22067628-2cb6-4c99-b404-7d4ccd6231df-kube-api-access-qjpll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482454 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22067628-2cb6-4c99-b404-7d4ccd6231df-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482454 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482454 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482616 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482523 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482616 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482542 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.482616 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.482589 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.484571 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.484549 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22067628-2cb6-4c99-b404-7d4ccd6231df-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.484862 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.484845 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22067628-2cb6-4c99-b404-7d4ccd6231df-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.491004 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.490982 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpll\" (UniqueName: \"kubernetes.io/projected/22067628-2cb6-4c99-b404-7d4ccd6231df-kube-api-access-qjpll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z6z55\" (UID: \"22067628-2cb6-4c99-b404-7d4ccd6231df\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.535682 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.535623 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:13.671948 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:13.671920 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55"] Jul 06 13:00:13.673389 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:00:13.673357 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22067628_2cb6_4c99_b404_7d4ccd6231df.slice/crio-597f391a4dbc19e309d84f4495c62937fc147eb2edd0be299cd7e075faf61bf4 WatchSource:0}: Error finding container 597f391a4dbc19e309d84f4495c62937fc147eb2edd0be299cd7e075faf61bf4: Status 404 returned error can't find the container with id 597f391a4dbc19e309d84f4495c62937fc147eb2edd0be299cd7e075faf61bf4 Jul 06 13:00:14.032819 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:14.032775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" event={"ID":"22067628-2cb6-4c99-b404-7d4ccd6231df","Type":"ContainerStarted","Data":"597f391a4dbc19e309d84f4495c62937fc147eb2edd0be299cd7e075faf61bf4"} Jul 06 13:00:17.050973 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:17.050882 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" event={"ID":"da67aed7-44a3-45f2-95d7-0a44bae09e18","Type":"ContainerStarted","Data":"6d34112063e99c3e81c62e9d8a6dc9364fc330dbb55ff5d53ec4613ce62ef01c"} Jul 06 13:00:17.070214 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:17.070163 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" podStartSLOduration=1.261913999 podStartE2EDuration="17.070147521s" totalImagesPullingTime="15.808233522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:00:17.069609566 +0000 UTC m=+801.010450426" watchObservedRunningTime="2026-07-06 13:00:17.070147521 +0000 UTC m=+801.010988380" Jul 06 13:00:18.057514 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:18.057483 2568 generic.go:356] "Generic (PLEG): container finished" podID="da67aed7-44a3-45f2-95d7-0a44bae09e18" containerID="6d34112063e99c3e81c62e9d8a6dc9364fc330dbb55ff5d53ec4613ce62ef01c" exitCode=0 Jul 06 13:00:18.057919 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:18.057542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" event={"ID":"da67aed7-44a3-45f2-95d7-0a44bae09e18","Type":"ContainerDied","Data":"6d34112063e99c3e81c62e9d8a6dc9364fc330dbb55ff5d53ec4613ce62ef01c"} Jul 06 13:00:19.194659 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:19.194617 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" Jul 06 13:00:19.230783 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:19.230755 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w2qv\" (UniqueName: \"kubernetes.io/projected/da67aed7-44a3-45f2-95d7-0a44bae09e18-kube-api-access-2w2qv\") pod \"da67aed7-44a3-45f2-95d7-0a44bae09e18\" (UID: \"da67aed7-44a3-45f2-95d7-0a44bae09e18\") " Jul 06 13:00:19.232949 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:19.232923 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da67aed7-44a3-45f2-95d7-0a44bae09e18-kube-api-access-2w2qv" (OuterVolumeSpecName: "kube-api-access-2w2qv") pod "da67aed7-44a3-45f2-95d7-0a44bae09e18" (UID: "da67aed7-44a3-45f2-95d7-0a44bae09e18"). InnerVolumeSpecName "kube-api-access-2w2qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:00:19.332071 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:19.332010 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w2qv\" (UniqueName: \"kubernetes.io/projected/da67aed7-44a3-45f2-95d7-0a44bae09e18-kube-api-access-2w2qv\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:00:20.069527 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:20.069503 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" Jul 06 13:00:20.069720 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:20.069496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722380-x64h8" event={"ID":"da67aed7-44a3-45f2-95d7-0a44bae09e18","Type":"ContainerDied","Data":"dee6cb3c4a8e4fbf58013c3f9c4c7c9581d1b5fb75d5382b7cf48c80409f5a61"} Jul 06 13:00:20.069720 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:20.069666 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee6cb3c4a8e4fbf58013c3f9c4c7c9581d1b5fb75d5382b7cf48c80409f5a61" Jul 06 13:00:30.121965 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:30.121925 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" event={"ID":"22067628-2cb6-4c99-b404-7d4ccd6231df","Type":"ContainerStarted","Data":"d6fad2ba402566589f9329efd57a0903fc051bb9a402ec9487d88f70599ee26f"} Jul 06 13:00:30.142904 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:30.142856 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" podStartSLOduration=1.437394482 podStartE2EDuration="17.142842782s" totalImagesPullingTime="15.7054483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:00:30.141098267 +0000 UTC m=+814.081939135" watchObservedRunningTime="2026-07-06 13:00:30.142842782 +0000 UTC m=+814.083683640" Jul 06 13:00:33.536730 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:33.536686 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:33.537166 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:33.536791 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:33.549686 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:33.549664 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:34.152944 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:34.152916 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z6z55" Jul 06 13:00:50.272866 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.272828 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n"] Jul 06 13:00:50.273446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.273365 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da67aed7-44a3-45f2-95d7-0a44bae09e18" containerName="cleanup" Jul 06 13:00:50.273446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.273385 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da67aed7-44a3-45f2-95d7-0a44bae09e18" containerName="cleanup" Jul 06 13:00:50.273565 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.273481 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="da67aed7-44a3-45f2-95d7-0a44bae09e18" containerName="cleanup" Jul 06 13:00:50.279977 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.279949 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.282609 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.282589 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 13:00:50.385476 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.385449 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8ebc15-fde1-468d-afdd-7cd26b33d550-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.385655 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.385486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.385655 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.385519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.385655 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.385548 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.385790 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.385662 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.385790 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.385722 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/7e8ebc15-fde1-468d-afdd-7cd26b33d550-kube-api-access-w5krs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.425626 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.425586 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n"] Jul 06 13:00:50.486891 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.486866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/7e8ebc15-fde1-468d-afdd-7cd26b33d550-kube-api-access-w5krs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487062 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.486919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8ebc15-fde1-468d-afdd-7cd26b33d550-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487062 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.486950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487062 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.486968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487062 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.486990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487246 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.487069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487445 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.487421 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487445 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.487439 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.487566 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.487546 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.489486 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.489468 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8ebc15-fde1-468d-afdd-7cd26b33d550-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.489814 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.489796 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8ebc15-fde1-468d-afdd-7cd26b33d550-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.494991 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.494968 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/7e8ebc15-fde1-468d-afdd-7cd26b33d550-kube-api-access-w5krs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n\" (UID: \"7e8ebc15-fde1-468d-afdd-7cd26b33d550\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.589618 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.589529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:00:50.727266 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:50.727239 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n"] Jul 06 13:00:50.729339 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:00:50.729315 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8ebc15_fde1_468d_afdd_7cd26b33d550.slice/crio-4f1eb61d0234f6d297450814c45d01e865e4893c1803625d1f54cb2b47409d05 WatchSource:0}: Error finding container 4f1eb61d0234f6d297450814c45d01e865e4893c1803625d1f54cb2b47409d05: Status 404 returned error can't find the container with id 4f1eb61d0234f6d297450814c45d01e865e4893c1803625d1f54cb2b47409d05 Jul 06 13:00:51.231754 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:00:51.231722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" event={"ID":"7e8ebc15-fde1-468d-afdd-7cd26b33d550","Type":"ContainerStarted","Data":"4f1eb61d0234f6d297450814c45d01e865e4893c1803625d1f54cb2b47409d05"} Jul 06 13:01:07.904234 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:07.904189 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2"] Jul 06 13:01:07.910583 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:07.910565 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:07.913691 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:07.913675 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 13:01:07.919694 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:07.919673 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2"] Jul 06 13:01:08.050782 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.050744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.050782 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.050782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235086-9e05-44b4-b155-27ace499e6a0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.051007 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.050800 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.051007 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.050891 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.051007 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.050932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jht\" (UniqueName: \"kubernetes.io/projected/ec235086-9e05-44b4-b155-27ace499e6a0-kube-api-access-s4jht\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.051007 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.050976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152022 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.151986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152022 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jht\" (UniqueName: \"kubernetes.io/projected/ec235086-9e05-44b4-b155-27ace499e6a0-kube-api-access-s4jht\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152238 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152319 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152369 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235086-9e05-44b4-b155-27ace499e6a0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152369 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152356 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152454 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152373 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152550 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152534 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.152751 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.152730 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.154723 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.154661 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235086-9e05-44b4-b155-27ace499e6a0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.154988 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.154971 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235086-9e05-44b4-b155-27ace499e6a0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.161286 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.161261 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jht\" (UniqueName: \"kubernetes.io/projected/ec235086-9e05-44b4-b155-27ace499e6a0-kube-api-access-s4jht\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2\" (UID: \"ec235086-9e05-44b4-b155-27ace499e6a0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.219818 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.219770 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:08.362693 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:08.362664 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2"] Jul 06 13:01:08.364415 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:01:08.364383 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec235086_9e05_44b4_b155_27ace499e6a0.slice/crio-73f6a73fdd6837472ca13579313fa1737bd3cef5b9db2a7e46c0093babbd3c7e WatchSource:0}: Error finding container 73f6a73fdd6837472ca13579313fa1737bd3cef5b9db2a7e46c0093babbd3c7e: Status 404 returned error can't find the container with id 73f6a73fdd6837472ca13579313fa1737bd3cef5b9db2a7e46c0093babbd3c7e Jul 06 13:01:09.327164 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:09.327129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" event={"ID":"ec235086-9e05-44b4-b155-27ace499e6a0","Type":"ContainerStarted","Data":"73f6a73fdd6837472ca13579313fa1737bd3cef5b9db2a7e46c0093babbd3c7e"} Jul 06 13:01:13.351001 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:13.350969 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" event={"ID":"7e8ebc15-fde1-468d-afdd-7cd26b33d550","Type":"ContainerStarted","Data":"f0096985f414c993a55a183437d20491986ed6884f9d1c2538e57cdf8f546abd"} Jul 06 13:01:13.371650 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:13.371584 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" podStartSLOduration=1.088531496 podStartE2EDuration="23.371568134s" totalImagesPullingTime="22.283036638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:01:13.370947271 +0000 UTC m=+857.311788143" watchObservedRunningTime="2026-07-06 13:01:13.371568134 +0000 UTC m=+857.312409030" Jul 06 13:01:20.590334 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:20.590299 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:01:20.590791 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:20.590358 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:01:20.602643 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:20.602608 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:01:21.404768 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:21.404737 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n" Jul 06 13:01:27.427961 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:27.427924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" event={"ID":"ec235086-9e05-44b4-b155-27ace499e6a0","Type":"ContainerStarted","Data":"54b8c3812dd6bf584dc97f894547553c850406db35d2e77ca552cf9ebd16c34a"} Jul 06 13:01:27.448985 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:27.448939 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" podStartSLOduration=1.875853181 podStartE2EDuration="20.448925645s" totalImagesPullingTime="18.573072464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:01:27.446699243 +0000 UTC m=+871.387540105" watchObservedRunningTime="2026-07-06 13:01:27.448925645 +0000 UTC m=+871.389766503" Jul 06 13:01:28.220828 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:28.220788 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:28.220828 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:28.220833 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:28.233659 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:28.233608 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:28.444421 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:28.444395 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2" Jul 06 13:01:56.555563 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:56.555474 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:01:56.556237 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:01:56.555587 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:06:56.598492 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:06:56.598458 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:06:56.600909 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:06:56.600688 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:08:10.825053 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:10.824971 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm"] Jul 06 13:08:10.825581 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:10.825160 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" podUID="e3c1ef71-6fa0-4080-8e89-68a3c030f686" containerName="manager" containerID="cri-o://c2d8f8680959f5ae1eeffdb289ca95fe01c677727d8756c5201b34a8c1ff8e3e" gracePeriod=10 Jul 06 13:08:14.596232 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:14.596150 2568 generic.go:356] "Generic (PLEG): container finished" podID="e3c1ef71-6fa0-4080-8e89-68a3c030f686" containerID="c2d8f8680959f5ae1eeffdb289ca95fe01c677727d8756c5201b34a8c1ff8e3e" exitCode=0 Jul 06 13:08:14.596232 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:14.596187 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" event={"ID":"e3c1ef71-6fa0-4080-8e89-68a3c030f686","Type":"ContainerDied","Data":"c2d8f8680959f5ae1eeffdb289ca95fe01c677727d8756c5201b34a8c1ff8e3e"} Jul 06 13:08:14.974915 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:14.974888 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 13:08:15.059956 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.059930 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf7z2\" (UniqueName: \"kubernetes.io/projected/e3c1ef71-6fa0-4080-8e89-68a3c030f686-kube-api-access-kf7z2\") pod \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " Jul 06 13:08:15.060113 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.059981 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3c1ef71-6fa0-4080-8e89-68a3c030f686-extensions-socket-volume\") pod \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\" (UID: \"e3c1ef71-6fa0-4080-8e89-68a3c030f686\") " Jul 06 13:08:15.060384 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.060359 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c1ef71-6fa0-4080-8e89-68a3c030f686-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e3c1ef71-6fa0-4080-8e89-68a3c030f686" (UID: "e3c1ef71-6fa0-4080-8e89-68a3c030f686"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:08:15.062042 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.062012 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c1ef71-6fa0-4080-8e89-68a3c030f686-kube-api-access-kf7z2" (OuterVolumeSpecName: "kube-api-access-kf7z2") pod "e3c1ef71-6fa0-4080-8e89-68a3c030f686" (UID: "e3c1ef71-6fa0-4080-8e89-68a3c030f686"). InnerVolumeSpecName "kube-api-access-kf7z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:08:15.161412 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.161346 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7z2\" (UniqueName: \"kubernetes.io/projected/e3c1ef71-6fa0-4080-8e89-68a3c030f686-kube-api-access-kf7z2\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:08:15.161412 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.161371 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3c1ef71-6fa0-4080-8e89-68a3c030f686-extensions-socket-volume\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:08:15.603051 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.603013 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" event={"ID":"e3c1ef71-6fa0-4080-8e89-68a3c030f686","Type":"ContainerDied","Data":"d6e9bfff3422e60947ffe24e420792d51a01f8fab5e0cf14f4c7e02b3f4ffa90"} Jul 06 13:08:15.603439 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.603068 2568 scope.go:117] "RemoveContainer" containerID="c2d8f8680959f5ae1eeffdb289ca95fe01c677727d8756c5201b34a8c1ff8e3e" Jul 06 13:08:15.603439 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.603030 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm" Jul 06 13:08:15.628879 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.628849 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm"] Jul 06 13:08:15.638282 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:15.638260 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-722qm"] Jul 06 13:08:16.590609 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:08:16.590577 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3c1ef71-6fa0-4080-8e89-68a3c030f686" path="/var/lib/kubelet/pods/e3c1ef71-6fa0-4080-8e89-68a3c030f686/volumes" Jul 06 13:09:11.955770 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:11.955736 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr"] Jul 06 13:09:11.956830 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:11.956117 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c1ef71-6fa0-4080-8e89-68a3c030f686" containerName="manager" Jul 06 13:09:11.956830 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:11.956136 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c1ef71-6fa0-4080-8e89-68a3c030f686" containerName="manager" Jul 06 13:09:11.956830 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:11.956227 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c1ef71-6fa0-4080-8e89-68a3c030f686" containerName="manager" Jul 06 13:09:11.959764 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:11.959745 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:11.963049 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:11.963032 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6mvwm\"" Jul 06 13:09:11.984085 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:11.984059 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr"] Jul 06 13:09:12.025610 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.025581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b3365e-a347-45fd-95ae-f16d30afc89e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xl8cr\" (UID: \"06b3365e-a347-45fd-95ae-f16d30afc89e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.025763 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.025625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdrm\" (UniqueName: \"kubernetes.io/projected/06b3365e-a347-45fd-95ae-f16d30afc89e-kube-api-access-mxdrm\") pod \"kuadrant-operator-controller-manager-86878c769f-xl8cr\" (UID: \"06b3365e-a347-45fd-95ae-f16d30afc89e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.126995 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.126970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdrm\" (UniqueName: \"kubernetes.io/projected/06b3365e-a347-45fd-95ae-f16d30afc89e-kube-api-access-mxdrm\") pod \"kuadrant-operator-controller-manager-86878c769f-xl8cr\" (UID: \"06b3365e-a347-45fd-95ae-f16d30afc89e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.127120 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.127045 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b3365e-a347-45fd-95ae-f16d30afc89e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xl8cr\" (UID: \"06b3365e-a347-45fd-95ae-f16d30afc89e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.127366 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.127351 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b3365e-a347-45fd-95ae-f16d30afc89e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xl8cr\" (UID: \"06b3365e-a347-45fd-95ae-f16d30afc89e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.137081 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.137061 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdrm\" (UniqueName: \"kubernetes.io/projected/06b3365e-a347-45fd-95ae-f16d30afc89e-kube-api-access-mxdrm\") pod \"kuadrant-operator-controller-manager-86878c769f-xl8cr\" (UID: \"06b3365e-a347-45fd-95ae-f16d30afc89e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.269997 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.269935 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.422905 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.422846 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr"] Jul 06 13:09:12.425427 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:09:12.425402 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b3365e_a347_45fd_95ae_f16d30afc89e.slice/crio-98a818d58bf1a17ded368c86d9027397cb835b9421b6d48f401da6c3dc6c06e6 WatchSource:0}: Error finding container 98a818d58bf1a17ded368c86d9027397cb835b9421b6d48f401da6c3dc6c06e6: Status 404 returned error can't find the container with id 98a818d58bf1a17ded368c86d9027397cb835b9421b6d48f401da6c3dc6c06e6 Jul 06 13:09:12.891010 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.890973 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" event={"ID":"06b3365e-a347-45fd-95ae-f16d30afc89e","Type":"ContainerStarted","Data":"f0a1cf9782315a1d87f886d0015af5ccf4be5a6a82633e0f34e36132c83e3108"} Jul 06 13:09:12.891211 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.891057 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" event={"ID":"06b3365e-a347-45fd-95ae-f16d30afc89e","Type":"ContainerStarted","Data":"98a818d58bf1a17ded368c86d9027397cb835b9421b6d48f401da6c3dc6c06e6"} Jul 06 13:09:12.891211 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.891088 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:09:12.914714 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:12.914661 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" podStartSLOduration=1.9146189310000001 podStartE2EDuration="1.914618931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:09:12.911238828 +0000 UTC m=+1336.852079689" watchObservedRunningTime="2026-07-06 13:09:12.914618931 +0000 UTC m=+1336.855459794" Jul 06 13:09:23.899358 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:09:23.899284 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xl8cr" Jul 06 13:11:56.632326 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:11:56.632215 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:11:56.636393 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:11:56.636235 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:15:26.245546 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.245457 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t"] Jul 06 13:15:26.249788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.249766 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.253526 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.253499 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f2d3593a-openshift-default-dockercfg-b2gnb\"" Jul 06 13:15:26.279753 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.279722 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t"] Jul 06 13:15:26.322219 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-credential-socket\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322219 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hs5h\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-kube-api-access-6hs5h\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322449 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322242 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-envoy\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322449 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32647024-9fd3-4f25-96cc-2472861a476b-istio-podinfo\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322449 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-data\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322449 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-socket\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322449 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322396 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32647024-9fd3-4f25-96cc-2472861a476b-istiod-ca-cert\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322449 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-certs\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.322673 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.322461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-istio-token\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423018 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.422991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-credential-socket\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423201 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hs5h\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-kube-api-access-6hs5h\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423201 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-envoy\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423201 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32647024-9fd3-4f25-96cc-2472861a476b-istio-podinfo\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423368 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423261 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-data\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423368 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-socket\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423368 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32647024-9fd3-4f25-96cc-2472861a476b-istiod-ca-cert\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423533 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-certs\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423533 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-istio-token\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423533 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423437 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-credential-socket\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423725 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423623 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-data\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423786 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-socket\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.423899 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.423804 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-certs\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.424067 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.424034 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32647024-9fd3-4f25-96cc-2472861a476b-istiod-ca-cert\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.425459 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.425440 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-envoy\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.425728 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.425711 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32647024-9fd3-4f25-96cc-2472861a476b-istio-podinfo\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.441078 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.441053 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-istio-token\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.441400 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.441384 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hs5h\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-kube-api-access-6hs5h\") pod \"e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.560770 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.560682 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:26.705310 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.705280 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t"] Jul 06 13:15:26.705436 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:15:26.705326 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32647024_9fd3_4f25_96cc_2472861a476b.slice/crio-32903b31d05129f22337d56aecf7103f72ae33e00e939d43492181827a75d8d5 WatchSource:0}: Error finding container 32903b31d05129f22337d56aecf7103f72ae33e00e939d43492181827a75d8d5: Status 404 returned error can't find the container with id 32903b31d05129f22337d56aecf7103f72ae33e00e939d43492181827a75d8d5 Jul 06 13:15:26.707448 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.707419 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:26.707534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.707481 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:26.707571 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.707546 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:26.842538 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.842445 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" event={"ID":"32647024-9fd3-4f25-96cc-2472861a476b","Type":"ContainerStarted","Data":"fd554b13802d7b3779bc5590e5ff64180fa224d4223d7649df981624acbe47dc"} Jul 06 13:15:26.842538 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.842506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" event={"ID":"32647024-9fd3-4f25-96cc-2472861a476b","Type":"ContainerStarted","Data":"32903b31d05129f22337d56aecf7103f72ae33e00e939d43492181827a75d8d5"} Jul 06 13:15:26.875228 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:26.875176 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" podStartSLOduration=0.875161215 podStartE2EDuration="875.161215ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:15:26.871807757 +0000 UTC m=+1710.812648639" watchObservedRunningTime="2026-07-06 13:15:26.875161215 +0000 UTC m=+1710.816002073" Jul 06 13:15:27.561714 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:27.561675 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:27.566719 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:27.566695 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:27.847366 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:27.847288 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:27.848286 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:27.848269 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:32.730715 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:32.730680 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t"] Jul 06 13:15:32.731142 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:32.730856 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" podUID="32647024-9fd3-4f25-96cc-2472861a476b" containerName="istio-proxy" containerID="cri-o://fd554b13802d7b3779bc5590e5ff64180fa224d4223d7649df981624acbe47dc" gracePeriod=30 Jul 06 13:15:38.915534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:38.915495 2568 generic.go:356] "Generic (PLEG): container finished" podID="32647024-9fd3-4f25-96cc-2472861a476b" containerID="fd554b13802d7b3779bc5590e5ff64180fa224d4223d7649df981624acbe47dc" exitCode=0 Jul 06 13:15:38.915924 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:38.915583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" event={"ID":"32647024-9fd3-4f25-96cc-2472861a476b","Type":"ContainerDied","Data":"fd554b13802d7b3779bc5590e5ff64180fa224d4223d7649df981624acbe47dc"} Jul 06 13:15:38.994288 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:38.994239 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:39.136453 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136422 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32647024-9fd3-4f25-96cc-2472861a476b-istio-podinfo\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136653 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136489 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-certs\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136653 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136522 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hs5h\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-kube-api-access-6hs5h\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136653 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136545 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-data\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136653 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136573 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-envoy\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136653 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136618 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-istio-token\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136914 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136681 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32647024-9fd3-4f25-96cc-2472861a476b-istiod-ca-cert\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136914 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136712 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-socket\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136914 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136787 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-credential-socket\") pod \"32647024-9fd3-4f25-96cc-2472861a476b\" (UID: \"32647024-9fd3-4f25-96cc-2472861a476b\") " Jul 06 13:15:39.136914 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.136839 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:39.137139 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.137101 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32647024-9fd3-4f25-96cc-2472861a476b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:15:39.137206 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.137135 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.137600 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.137469 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:39.137753 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.137528 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-data" (OuterVolumeSpecName: "istio-data") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:39.137753 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.137710 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:39.139166 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.139139 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:39.139259 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.139195 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-istio-token" (OuterVolumeSpecName: "istio-token") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:15:39.139394 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.139377 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32647024-9fd3-4f25-96cc-2472861a476b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:15:39.139448 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.139389 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-kube-api-access-6hs5h" (OuterVolumeSpecName: "kube-api-access-6hs5h") pod "32647024-9fd3-4f25-96cc-2472861a476b" (UID: "32647024-9fd3-4f25-96cc-2472861a476b"). InnerVolumeSpecName "kube-api-access-6hs5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:15:39.238162 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238131 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32647024-9fd3-4f25-96cc-2472861a476b-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.238162 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238156 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.238162 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238165 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.238355 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238177 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32647024-9fd3-4f25-96cc-2472861a476b-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.238355 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238186 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hs5h\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-kube-api-access-6hs5h\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.238355 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238195 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.238355 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238203 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32647024-9fd3-4f25-96cc-2472861a476b-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.238355 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.238210 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32647024-9fd3-4f25-96cc-2472861a476b-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:39.676031 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.675996 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb"] Jul 06 13:15:39.676534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.676518 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32647024-9fd3-4f25-96cc-2472861a476b" containerName="istio-proxy" Jul 06 13:15:39.676592 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.676537 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="32647024-9fd3-4f25-96cc-2472861a476b" containerName="istio-proxy" Jul 06 13:15:39.676686 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.676673 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="32647024-9fd3-4f25-96cc-2472861a476b" containerName="istio-proxy" Jul 06 13:15:39.680083 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.680067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.683158 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.683137 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aedba5e8-openshift-default-dockercfg-45nzb\"" Jul 06 13:15:39.732469 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.732441 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb"] Jul 06 13:15:39.843866 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.843827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-socket\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844070 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.843874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrv4\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-kube-api-access-bmrv4\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844070 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.843916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-token\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844070 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.843975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-podinfo\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844070 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.844024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-credential-socket\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844264 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.844065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-data\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844264 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.844115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-certs\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844264 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.844134 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-envoy\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.844264 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.844170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istiod-ca-cert\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.923115 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.923084 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" Jul 06 13:15:39.923115 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.923088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t" event={"ID":"32647024-9fd3-4f25-96cc-2472861a476b","Type":"ContainerDied","Data":"32903b31d05129f22337d56aecf7103f72ae33e00e939d43492181827a75d8d5"} Jul 06 13:15:39.923576 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.923159 2568 scope.go:117] "RemoveContainer" containerID="fd554b13802d7b3779bc5590e5ff64180fa224d4223d7649df981624acbe47dc" Jul 06 13:15:39.944961 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.944936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-certs\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945082 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.944968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-envoy\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945082 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.944998 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istiod-ca-cert\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945082 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-socket\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945082 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrv4\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-kube-api-access-bmrv4\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945082 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-token\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945329 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945109 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-podinfo\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945329 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945148 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-credential-socket\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945329 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945293 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-data\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945487 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945328 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-certs\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945487 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945435 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-credential-socket\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945605 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945497 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-socket\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945850 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945679 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-data\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.945906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.945890 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istiod-ca-cert\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.947321 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.947292 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-envoy\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.947750 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.947721 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-podinfo\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.951377 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.951357 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t"] Jul 06 13:15:39.962114 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.962086 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-token\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.964082 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.964053 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f2d3593a-openshift-default-74764d5857-fpx7t"] Jul 06 13:15:39.970751 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.970728 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrv4\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-kube-api-access-bmrv4\") pod \"e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:39.990425 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:39.990401 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:40.129939 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.129914 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb"] Jul 06 13:15:40.130887 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:15:40.130861 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61da5159_c92e_498c_9ed5_9fa55fb9d5ec.slice/crio-9c5117b616299b36fc79d7566c97380d6b0c2a133d93a7640f7f3dbf24d6b489 WatchSource:0}: Error finding container 9c5117b616299b36fc79d7566c97380d6b0c2a133d93a7640f7f3dbf24d6b489: Status 404 returned error can't find the container with id 9c5117b616299b36fc79d7566c97380d6b0c2a133d93a7640f7f3dbf24d6b489 Jul 06 13:15:40.132994 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.132967 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:40.133070 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.133048 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:40.133112 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.133079 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:40.590997 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.590891 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32647024-9fd3-4f25-96cc-2472861a476b" path="/var/lib/kubelet/pods/32647024-9fd3-4f25-96cc-2472861a476b/volumes" Jul 06 13:15:40.931119 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.931087 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" event={"ID":"61da5159-c92e-498c-9ed5-9fa55fb9d5ec","Type":"ContainerStarted","Data":"86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb"} Jul 06 13:15:40.931500 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.931136 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" event={"ID":"61da5159-c92e-498c-9ed5-9fa55fb9d5ec","Type":"ContainerStarted","Data":"9c5117b616299b36fc79d7566c97380d6b0c2a133d93a7640f7f3dbf24d6b489"} Jul 06 13:15:40.966226 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.966165 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" podStartSLOduration=1.966145134 podStartE2EDuration="1.966145134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:15:40.96152399 +0000 UTC m=+1724.902364862" watchObservedRunningTime="2026-07-06 13:15:40.966145134 +0000 UTC m=+1724.906985994" Jul 06 13:15:40.991080 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:40.991052 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:41.995366 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:41.995339 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:42.941965 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:42.941933 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:42.942822 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:42.942800 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:52.011236 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:52.011198 2568 generic.go:356] "Generic (PLEG): container finished" podID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerID="86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb" exitCode=0 Jul 06 13:15:52.011654 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:52.011272 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" event={"ID":"61da5159-c92e-498c-9ed5-9fa55fb9d5ec","Type":"ContainerDied","Data":"86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb"} Jul 06 13:15:52.011700 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:52.011669 2568 scope.go:117] "RemoveContainer" containerID="86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb" Jul 06 13:15:52.012601 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:52.012574 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:52.012673 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:52.012655 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:52.012714 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:52.012687 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:52.105583 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:52.105552 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb"] Jul 06 13:15:53.018991 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:53.018952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" event={"ID":"61da5159-c92e-498c-9ed5-9fa55fb9d5ec","Type":"ContainerStarted","Data":"90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2"} Jul 06 13:15:53.019404 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:53.019018 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerName="istio-proxy" containerID="cri-o://90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2" gracePeriod=30 Jul 06 13:15:57.942700 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:57.942648 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:58.274141 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.274117 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:58.314789 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.314761 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-credential-socket\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.314966 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.314819 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-token\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.314966 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.314864 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-podinfo\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.314966 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.314904 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmrv4\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-kube-api-access-bmrv4\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.314966 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.314927 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istiod-ca-cert\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.314966 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.314963 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-envoy\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.315223 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.314990 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-certs\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.315223 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.315025 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-socket\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.315223 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.315028 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:58.315223 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.315049 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-data\") pod \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\" (UID: \"61da5159-c92e-498c-9ed5-9fa55fb9d5ec\") " Jul 06 13:15:58.315418 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.315318 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.315418 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.315395 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:58.315577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.315448 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:15:58.315718 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.315687 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:58.316484 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.316455 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:58.317237 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.317215 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:15:58.317437 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.317413 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-kube-api-access-bmrv4" (OuterVolumeSpecName: "kube-api-access-bmrv4") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "kube-api-access-bmrv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:15:58.317498 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.317416 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:15:58.317674 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.317651 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61da5159-c92e-498c-9ed5-9fa55fb9d5ec" (UID: "61da5159-c92e-498c-9ed5-9fa55fb9d5ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:15:58.416224 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416193 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrv4\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-kube-api-access-bmrv4\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.416224 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416222 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.416224 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416231 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.416436 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416239 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.416436 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416247 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.416436 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416255 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.416436 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416263 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.416436 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.416270 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61da5159-c92e-498c-9ed5-9fa55fb9d5ec-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:15:58.804378 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.804346 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d"] Jul 06 13:15:58.804833 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.804799 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerName="istio-proxy" Jul 06 13:15:58.804833 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.804823 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerName="istio-proxy" Jul 06 13:15:58.805001 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.804842 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerName="istio-proxy" Jul 06 13:15:58.805001 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.804853 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerName="istio-proxy" Jul 06 13:15:58.805001 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.804931 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerName="istio-proxy" Jul 06 13:15:58.805001 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.804940 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerName="istio-proxy" Jul 06 13:15:58.808192 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.808176 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.810533 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.810514 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3593f9e9-openshift-default-dockercfg-rt2t5\"" Jul 06 13:15:58.834159 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.834133 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d"] Jul 06 13:15:58.921067 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.920981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-credential-socket\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921067 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921019 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-certs\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921067 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921059 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-envoy\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921084 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74069e18-dbcc-4ae5-810f-1120002273d9-istio-podinfo\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-istio-token\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-socket\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-data\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74069e18-dbcc-4ae5-810f-1120002273d9-istiod-ca-cert\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:58.921438 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:58.921298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvgr\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-kube-api-access-cbvgr\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022290 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022255 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-envoy\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022290 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74069e18-dbcc-4ae5-810f-1120002273d9-istio-podinfo\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-istio-token\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-socket\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-data\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022570 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74069e18-dbcc-4ae5-810f-1120002273d9-istiod-ca-cert\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvgr\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-kube-api-access-cbvgr\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-credential-socket\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.022788 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-certs\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.023146 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022847 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-socket\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.023146 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.022909 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-data\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.023146 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.023002 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-credential-socket\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.023146 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.023021 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-certs\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.023484 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.023459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74069e18-dbcc-4ae5-810f-1120002273d9-istiod-ca-cert\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.024671 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.024643 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-envoy\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.024940 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.024925 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74069e18-dbcc-4ae5-810f-1120002273d9-istio-podinfo\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.031803 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.031781 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-istio-token\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.032236 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.032215 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvgr\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-kube-api-access-cbvgr\") pod \"e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.056358 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.056327 2568 generic.go:356] "Generic (PLEG): container finished" podID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" containerID="90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2" exitCode=0 Jul 06 13:15:59.056459 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.056375 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" Jul 06 13:15:59.056459 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.056375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" event={"ID":"61da5159-c92e-498c-9ed5-9fa55fb9d5ec","Type":"ContainerDied","Data":"90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2"} Jul 06 13:15:59.056534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.056498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb" event={"ID":"61da5159-c92e-498c-9ed5-9fa55fb9d5ec","Type":"ContainerDied","Data":"9c5117b616299b36fc79d7566c97380d6b0c2a133d93a7640f7f3dbf24d6b489"} Jul 06 13:15:59.056534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.056523 2568 scope.go:117] "RemoveContainer" containerID="90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2" Jul 06 13:15:59.066399 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.066384 2568 scope.go:117] "RemoveContainer" containerID="86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb" Jul 06 13:15:59.075481 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.075463 2568 scope.go:117] "RemoveContainer" containerID="90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2" Jul 06 13:15:59.075843 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:15:59.075816 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2\": container with ID starting with 90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2 not found: ID does not exist" containerID="90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2" Jul 06 13:15:59.075948 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.075851 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2"} err="failed to get container status \"90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2\": rpc error: code = NotFound desc = could not find container \"90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2\": container with ID starting with 90bd96d5568497b4ee434c61caa473d6fad617beb9ddb6cb75dee74197c707e2 not found: ID does not exist" Jul 06 13:15:59.075948 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.075874 2568 scope.go:117] "RemoveContainer" containerID="86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb" Jul 06 13:15:59.076312 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:15:59.076287 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb\": container with ID starting with 86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb not found: ID does not exist" containerID="86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb" Jul 06 13:15:59.076417 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.076318 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb"} err="failed to get container status \"86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb\": rpc error: code = NotFound desc = could not find container \"86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb\": container with ID starting with 86f3cc45e83283245257422b0ecd7461b54d3435e9b9282184512657f54a7efb not found: ID does not exist" Jul 06 13:15:59.078747 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.078726 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb"] Jul 06 13:15:59.093165 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.093145 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aedba5e8-openshift-default-7f4df4664f-4c2vb"] Jul 06 13:15:59.121274 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.121255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:15:59.262821 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.262796 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d"] Jul 06 13:15:59.265718 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:15:59.265688 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74069e18_dbcc_4ae5_810f_1120002273d9.slice/crio-4d95ccf53fc1b98635ea5e3fbc05a44319c1428e76e1a740c3bef2d35fc6ec7f WatchSource:0}: Error finding container 4d95ccf53fc1b98635ea5e3fbc05a44319c1428e76e1a740c3bef2d35fc6ec7f: Status 404 returned error can't find the container with id 4d95ccf53fc1b98635ea5e3fbc05a44319c1428e76e1a740c3bef2d35fc6ec7f Jul 06 13:15:59.268012 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.267980 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:59.268104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.268063 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:15:59.268104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:15:59.268095 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:16:00.065599 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:00.065561 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" event={"ID":"74069e18-dbcc-4ae5-810f-1120002273d9","Type":"ContainerStarted","Data":"4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7"} Jul 06 13:16:00.066092 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:00.065644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" event={"ID":"74069e18-dbcc-4ae5-810f-1120002273d9","Type":"ContainerStarted","Data":"4d95ccf53fc1b98635ea5e3fbc05a44319c1428e76e1a740c3bef2d35fc6ec7f"} Jul 06 13:16:00.091712 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:00.091658 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" podStartSLOduration=2.091623364 podStartE2EDuration="2.091623364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:16:00.090456696 +0000 UTC m=+1744.031297562" watchObservedRunningTime="2026-07-06 13:16:00.091623364 +0000 UTC m=+1744.032464224" Jul 06 13:16:00.122335 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:00.122308 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:16:00.127148 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:00.127129 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:16:00.593509 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:00.593475 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61da5159-c92e-498c-9ed5-9fa55fb9d5ec" path="/var/lib/kubelet/pods/61da5159-c92e-498c-9ed5-9fa55fb9d5ec/volumes" Jul 06 13:16:01.071469 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:01.071436 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:16:01.072383 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:01.072364 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:16:04.597019 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:04.596969 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d"] Jul 06 13:16:04.597940 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:04.597883 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" podUID="74069e18-dbcc-4ae5-810f-1120002273d9" containerName="istio-proxy" containerID="cri-o://4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7" gracePeriod=30 Jul 06 13:16:11.770390 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.770368 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:16:11.840284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840210 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-envoy\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840245 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-credential-socket\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840284 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840278 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74069e18-dbcc-4ae5-810f-1120002273d9-istiod-ca-cert\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840515 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840313 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-data\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840515 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840370 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74069e18-dbcc-4ae5-810f-1120002273d9-istio-podinfo\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840515 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840501 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:16:11.840704 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840534 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-certs\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840704 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840575 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-socket\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840704 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840614 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-istio-token\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840704 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840667 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbvgr\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-kube-api-access-cbvgr\") pod \"74069e18-dbcc-4ae5-810f-1120002273d9\" (UID: \"74069e18-dbcc-4ae5-810f-1120002273d9\") " Jul 06 13:16:11.840896 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840784 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:16:11.840896 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.840789 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74069e18-dbcc-4ae5-810f-1120002273d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:16:11.841040 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.841023 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.841102 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.841046 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74069e18-dbcc-4ae5-810f-1120002273d9-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.841102 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.841060 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.841102 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.841063 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:16:11.841262 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.841198 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:16:11.842857 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.842826 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:16:11.843031 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.843010 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-kube-api-access-cbvgr" (OuterVolumeSpecName: "kube-api-access-cbvgr") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "kube-api-access-cbvgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:16:11.843142 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.843114 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:16:11.843233 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.843175 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74069e18-dbcc-4ae5-810f-1120002273d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74069e18-dbcc-4ae5-810f-1120002273d9" (UID: "74069e18-dbcc-4ae5-810f-1120002273d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:16:11.942318 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.942290 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.942318 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.942315 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.942318 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.942325 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74069e18-dbcc-4ae5-810f-1120002273d9-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.942671 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.942336 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74069e18-dbcc-4ae5-810f-1120002273d9-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.942671 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.942351 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:11.942671 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:11.942360 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvgr\" (UniqueName: \"kubernetes.io/projected/74069e18-dbcc-4ae5-810f-1120002273d9-kube-api-access-cbvgr\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:16:12.134489 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.134455 2568 generic.go:356] "Generic (PLEG): container finished" podID="74069e18-dbcc-4ae5-810f-1120002273d9" containerID="4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7" exitCode=0 Jul 06 13:16:12.134729 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.134505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" event={"ID":"74069e18-dbcc-4ae5-810f-1120002273d9","Type":"ContainerDied","Data":"4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7"} Jul 06 13:16:12.134729 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.134515 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" Jul 06 13:16:12.134729 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.134589 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d" event={"ID":"74069e18-dbcc-4ae5-810f-1120002273d9","Type":"ContainerDied","Data":"4d95ccf53fc1b98635ea5e3fbc05a44319c1428e76e1a740c3bef2d35fc6ec7f"} Jul 06 13:16:12.134729 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.134617 2568 scope.go:117] "RemoveContainer" containerID="4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7" Jul 06 13:16:12.144553 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.144535 2568 scope.go:117] "RemoveContainer" containerID="4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7" Jul 06 13:16:12.144811 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:16:12.144789 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7\": container with ID starting with 4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7 not found: ID does not exist" containerID="4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7" Jul 06 13:16:12.144868 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.144818 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7"} err="failed to get container status \"4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7\": rpc error: code = NotFound desc = could not find container \"4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7\": container with ID starting with 4865814721cd3e631fb97d962686cb575797e561806c191ca05595ea704472c7 not found: ID does not exist" Jul 06 13:16:12.165161 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.165132 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d"] Jul 06 13:16:12.186061 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.186025 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3593f9e9-openshift-default-896dd4795-ktw7d"] Jul 06 13:16:12.591364 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:12.591282 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74069e18-dbcc-4ae5-810f-1120002273d9" path="/var/lib/kubelet/pods/74069e18-dbcc-4ae5-810f-1120002273d9/volumes" Jul 06 13:16:56.666212 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:56.666175 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:16:56.671713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:16:56.671689 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:17:56.661870 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.661840 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48"] Jul 06 13:17:56.662372 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.662219 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74069e18-dbcc-4ae5-810f-1120002273d9" containerName="istio-proxy" Jul 06 13:17:56.662372 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.662231 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="74069e18-dbcc-4ae5-810f-1120002273d9" containerName="istio-proxy" Jul 06 13:17:56.662372 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.662296 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="74069e18-dbcc-4ae5-810f-1120002273d9" containerName="istio-proxy" Jul 06 13:17:56.665313 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.665294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.668463 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.668444 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0d2a555d-openshift-default-dockercfg-js2bg\"" Jul 06 13:17:56.714044 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.714015 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48"] Jul 06 13:17:56.747477 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-certs\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747643 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-socket\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747643 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-data\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747643 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c43138-deac-4146-8fb4-ca7ca29df9df-istiod-ca-cert\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747643 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-podinfo\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747643 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-credential-socket\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747855 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747652 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-token\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747855 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747722 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphg5\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-kube-api-access-hphg5\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.747855 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.747764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-envoy\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.848527 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-envoy\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.848737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-certs\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.848737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-socket\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.848737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-data\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.848737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c43138-deac-4146-8fb4-ca7ca29df9df-istiod-ca-cert\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.848737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848686 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-podinfo\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.848737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848713 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-credential-socket\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.849072 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-token\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.849072 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.848849 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphg5\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-kube-api-access-hphg5\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.849191 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.849080 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-socket\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.849191 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.849135 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-certs\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.849191 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.849171 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-data\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.849434 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.849414 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c43138-deac-4146-8fb4-ca7ca29df9df-istiod-ca-cert\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.849692 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.849492 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-credential-socket\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.851004 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.850983 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-envoy\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.851251 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.851234 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-podinfo\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.865792 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.865766 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-token\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.866793 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.866775 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphg5\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-kube-api-access-hphg5\") pod \"e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:56.976829 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:56.976764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:57.149128 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:17:57.149095 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c43138_deac_4146_8fb4_ca7ca29df9df.slice/crio-583cc0e15f2964b87f6cd0ed234e5eed36eeb5d7e60dcc48151d7492fb30dbe5 WatchSource:0}: Error finding container 583cc0e15f2964b87f6cd0ed234e5eed36eeb5d7e60dcc48151d7492fb30dbe5: Status 404 returned error can't find the container with id 583cc0e15f2964b87f6cd0ed234e5eed36eeb5d7e60dcc48151d7492fb30dbe5 Jul 06 13:17:57.149249 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:57.149223 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48"] Jul 06 13:17:57.151098 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:57.151066 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:17:57.151184 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:57.151127 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:17:57.151184 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:57.151157 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:17:57.700085 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:57.700052 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" event={"ID":"66c43138-deac-4146-8fb4-ca7ca29df9df","Type":"ContainerStarted","Data":"d2d2cd7c152d7f406ef9db942f765f778349fe0e85a04241ff41054e563f65fa"} Jul 06 13:17:57.700582 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:57.700110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" event={"ID":"66c43138-deac-4146-8fb4-ca7ca29df9df","Type":"ContainerStarted","Data":"583cc0e15f2964b87f6cd0ed234e5eed36eeb5d7e60dcc48151d7492fb30dbe5"} Jul 06 13:17:57.977433 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:57.977358 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:58.982468 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:58.982437 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:59.008435 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:59.008380 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" podStartSLOduration=3.008364818 podStartE2EDuration="3.008364818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:17:57.743471089 +0000 UTC m=+1861.684311992" watchObservedRunningTime="2026-07-06 13:17:59.008364818 +0000 UTC m=+1862.949205677" Jul 06 13:17:59.711593 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:59.711561 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:17:59.712603 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:17:59.712581 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:18:12.791885 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:12.791798 2568 generic.go:356] "Generic (PLEG): container finished" podID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerID="d2d2cd7c152d7f406ef9db942f765f778349fe0e85a04241ff41054e563f65fa" exitCode=0 Jul 06 13:18:12.791885 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:12.791877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" event={"ID":"66c43138-deac-4146-8fb4-ca7ca29df9df","Type":"ContainerDied","Data":"d2d2cd7c152d7f406ef9db942f765f778349fe0e85a04241ff41054e563f65fa"} Jul 06 13:18:12.792318 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:12.792275 2568 scope.go:117] "RemoveContainer" containerID="d2d2cd7c152d7f406ef9db942f765f778349fe0e85a04241ff41054e563f65fa" Jul 06 13:18:12.793329 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:12.793301 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:12.793401 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:12.793388 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:12.793439 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:12.793425 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:13.801894 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:13.801862 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" event={"ID":"66c43138-deac-4146-8fb4-ca7ca29df9df","Type":"ContainerStarted","Data":"83b87a9bf3df76168d3d6d59399b4d7157101a3a3be309fffb847c5103f0d4c7"} Jul 06 13:18:13.977386 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:13.977342 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:18:13.979110 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:13.979076 2568 patch_prober.go:28] interesting pod/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 06 13:18:13.979278 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:13.979138 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 06 13:18:14.712189 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:14.712145 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:18:14.978280 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:14.978177 2568 patch_prober.go:28] interesting pod/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 06 13:18:14.978761 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:14.978290 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 06 13:18:15.978138 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:15.978105 2568 patch_prober.go:28] interesting pod/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 06 13:18:15.978454 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:15.978427 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 06 13:18:16.978504 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:16.978023 2568 patch_prober.go:28] interesting pod/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 06 13:18:16.978504 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:16.978123 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 06 13:18:17.981562 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:17.981490 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:18:17.982560 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:17.982540 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:18:20.713212 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:20.713175 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48"] Jul 06 13:18:20.713622 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:20.713405 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" containerID="cri-o://83b87a9bf3df76168d3d6d59399b4d7157101a3a3be309fffb847c5103f0d4c7" gracePeriod=30 Jul 06 13:18:25.868577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:25.868538 2568 generic.go:356] "Generic (PLEG): container finished" podID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerID="83b87a9bf3df76168d3d6d59399b4d7157101a3a3be309fffb847c5103f0d4c7" exitCode=0 Jul 06 13:18:25.869062 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:25.868608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" event={"ID":"66c43138-deac-4146-8fb4-ca7ca29df9df","Type":"ContainerDied","Data":"83b87a9bf3df76168d3d6d59399b4d7157101a3a3be309fffb847c5103f0d4c7"} Jul 06 13:18:25.869062 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:25.868703 2568 scope.go:117] "RemoveContainer" containerID="d2d2cd7c152d7f406ef9db942f765f778349fe0e85a04241ff41054e563f65fa" Jul 06 13:18:25.975918 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:25.975837 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:18:26.017299 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017265 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-token\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017299 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017303 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-data\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017516 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017335 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-certs\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017516 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017365 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c43138-deac-4146-8fb4-ca7ca29df9df-istiod-ca-cert\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017516 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017391 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-podinfo\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017516 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017435 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-socket\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017516 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017456 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-envoy\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017516 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017494 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-credential-socket\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017850 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017524 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hphg5\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-kube-api-access-hphg5\") pod \"66c43138-deac-4146-8fb4-ca7ca29df9df\" (UID: \"66c43138-deac-4146-8fb4-ca7ca29df9df\") " Jul 06 13:18:26.017850 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017789 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c43138-deac-4146-8fb4-ca7ca29df9df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:18:26.018113 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.017968 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:18:26.018113 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.018025 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:18:26.018363 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.018203 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:18:26.018363 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.018291 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-data" (OuterVolumeSpecName: "istio-data") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:18:26.019888 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.019855 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:18:26.020168 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.020150 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-token" (OuterVolumeSpecName: "istio-token") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:18:26.020327 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.020298 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:18:26.020370 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.020327 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-kube-api-access-hphg5" (OuterVolumeSpecName: "kube-api-access-hphg5") pod "66c43138-deac-4146-8fb4-ca7ca29df9df" (UID: "66c43138-deac-4146-8fb4-ca7ca29df9df"). InnerVolumeSpecName "kube-api-access-hphg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:18:26.119032 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.118998 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119032 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119027 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119032 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119036 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119269 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119046 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphg5\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-kube-api-access-hphg5\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119269 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119055 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119269 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119063 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119269 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119070 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c43138-deac-4146-8fb4-ca7ca29df9df-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119269 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119078 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c43138-deac-4146-8fb4-ca7ca29df9df-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.119269 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.119085 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c43138-deac-4146-8fb4-ca7ca29df9df-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:18:26.875165 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.875066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" event={"ID":"66c43138-deac-4146-8fb4-ca7ca29df9df","Type":"ContainerDied","Data":"583cc0e15f2964b87f6cd0ed234e5eed36eeb5d7e60dcc48151d7492fb30dbe5"} Jul 06 13:18:26.875165 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.875127 2568 scope.go:117] "RemoveContainer" containerID="83b87a9bf3df76168d3d6d59399b4d7157101a3a3be309fffb847c5103f0d4c7" Jul 06 13:18:26.875733 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.875083 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48" Jul 06 13:18:26.897324 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.897299 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48"] Jul 06 13:18:26.902772 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:26.902752 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0d2a555d-openshift-default-758fccb899-vbv48"] Jul 06 13:18:28.591149 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:28.591116 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" path="/var/lib/kubelet/pods/66c43138-deac-4146-8fb4-ca7ca29df9df/volumes" Jul 06 13:18:41.702697 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.702669 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q"] Jul 06 13:18:41.703151 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.703004 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" Jul 06 13:18:41.703151 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.703015 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" Jul 06 13:18:41.703151 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.703025 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" Jul 06 13:18:41.703151 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.703030 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" Jul 06 13:18:41.703151 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.703109 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" Jul 06 13:18:41.703151 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.703116 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c43138-deac-4146-8fb4-ca7ca29df9df" containerName="istio-proxy" Jul 06 13:18:41.706157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.706137 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.712168 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.712147 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-69a5e1cc-openshift-default-dockercfg-bjj4s\"" Jul 06 13:18:41.727571 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.727547 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q"] Jul 06 13:18:41.756190 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-credential-socket\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756294 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-socket\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756294 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-data\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756395 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-istio-token\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756395 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-certs\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756395 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f30a118-1479-4863-8c65-93fc24343143-istiod-ca-cert\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756526 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-envoy\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756526 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756473 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwfg\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-kube-api-access-vjwfg\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.756619 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.756539 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f30a118-1479-4863-8c65-93fc24343143-istio-podinfo\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.857866 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.857837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-istio-token\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.857866 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.857880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-certs\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858136 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.857896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f30a118-1479-4863-8c65-93fc24343143-istiod-ca-cert\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858136 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.857919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-envoy\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858136 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.857938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwfg\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-kube-api-access-vjwfg\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858136 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.857986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f30a118-1479-4863-8c65-93fc24343143-istio-podinfo\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858136 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-credential-socket\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858136 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-socket\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858136 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858109 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-data\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858567 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858358 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-certs\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858567 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858445 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-data\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858567 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858523 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-credential-socket\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858567 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858533 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-socket\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.858808 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.858731 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f30a118-1479-4863-8c65-93fc24343143-istiod-ca-cert\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.860988 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.860965 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-envoy\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.861137 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.861121 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f30a118-1479-4863-8c65-93fc24343143-istio-podinfo\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.878111 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.878084 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-istio-token\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:41.879029 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:41.879011 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwfg\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-kube-api-access-vjwfg\") pod \"e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:42.016984 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.016901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:42.178222 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.178197 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q"] Jul 06 13:18:42.180515 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:18:42.180486 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f30a118_1479_4863_8c65_93fc24343143.slice/crio-b85240a70792d803b9d5a4d59458427f2d2113a831a0e8326daaf30154d004b6 WatchSource:0}: Error finding container b85240a70792d803b9d5a4d59458427f2d2113a831a0e8326daaf30154d004b6: Status 404 returned error can't find the container with id b85240a70792d803b9d5a4d59458427f2d2113a831a0e8326daaf30154d004b6 Jul 06 13:18:42.182698 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.182618 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:42.182789 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.182717 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:42.182789 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.182754 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:42.961871 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.961825 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" event={"ID":"5f30a118-1479-4863-8c65-93fc24343143","Type":"ContainerStarted","Data":"6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566"} Jul 06 13:18:42.962262 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.961896 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" event={"ID":"5f30a118-1479-4863-8c65-93fc24343143","Type":"ContainerStarted","Data":"b85240a70792d803b9d5a4d59458427f2d2113a831a0e8326daaf30154d004b6"} Jul 06 13:18:42.998601 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:42.998553 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" podStartSLOduration=1.9985377340000001 podStartE2EDuration="1.998537734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:18:42.996369952 +0000 UTC m=+1906.937210834" watchObservedRunningTime="2026-07-06 13:18:42.998537734 +0000 UTC m=+1906.939378592" Jul 06 13:18:43.017498 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:43.017472 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:43.022768 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:43.022747 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:43.967453 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:43.967419 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:43.968442 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:43.968424 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:18:58.969547 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:58.969511 2568 patch_prober.go:28] interesting pod/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 13:18:58.969963 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:58.969568 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 13:18:59.043398 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:59.043328 2568 generic.go:356] "Generic (PLEG): container finished" podID="5f30a118-1479-4863-8c65-93fc24343143" containerID="6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566" exitCode=0 Jul 06 13:18:59.043536 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:59.043408 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" event={"ID":"5f30a118-1479-4863-8c65-93fc24343143","Type":"ContainerDied","Data":"6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566"} Jul 06 13:18:59.043841 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:59.043819 2568 scope.go:117] "RemoveContainer" containerID="6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566" Jul 06 13:18:59.044951 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:59.044917 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:59.045063 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:59.045003 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:18:59.045063 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:18:59.045053 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:19:00.050768 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:00.050729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" event={"ID":"5f30a118-1479-4863-8c65-93fc24343143","Type":"ContainerStarted","Data":"04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e"} Jul 06 13:19:01.017750 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:01.017713 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:19:01.017937 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:01.017918 2568 patch_prober.go:28] interesting pod/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 13:19:01.017991 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:01.017972 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 13:19:02.017501 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:02.017466 2568 patch_prober.go:28] interesting pod/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 13:19:02.017924 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:02.017519 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 13:19:03.017624 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:03.017586 2568 patch_prober.go:28] interesting pod/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 06 13:19:03.018070 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:03.017670 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 06 13:19:04.022002 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:04.021971 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:19:04.022458 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:04.022339 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:19:04.022929 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:19:04.022914 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:20:18.412599 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:18.412564 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q"] Jul 06 13:20:18.413125 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:18.412847 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" containerID="cri-o://04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e" gracePeriod=30 Jul 06 13:20:19.024687 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:19.024650 2568 patch_prober.go:28] interesting pod/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 13:20:19.024877 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:19.024720 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 13:20:23.672960 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.672886 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:20:23.768378 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768349 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-certs\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768553 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768391 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-data\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768553 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768441 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-credential-socket\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768553 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768462 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-socket\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768553 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768485 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f30a118-1479-4863-8c65-93fc24343143-istio-podinfo\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768553 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768521 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f30a118-1479-4863-8c65-93fc24343143-istiod-ca-cert\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768839 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768554 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-envoy\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768839 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768693 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-istio-token\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768839 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768720 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjwfg\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-kube-api-access-vjwfg\") pod \"5f30a118-1479-4863-8c65-93fc24343143\" (UID: \"5f30a118-1479-4863-8c65-93fc24343143\") " Jul 06 13:20:23.768987 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768685 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:20:23.769052 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.768795 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:20:23.769111 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.769083 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.769111 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.769105 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.769216 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.769149 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f30a118-1479-4863-8c65-93fc24343143-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:20:23.769616 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.769514 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:20:23.770276 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.770240 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:20:23.771119 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.771082 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-kube-api-access-vjwfg" (OuterVolumeSpecName: "kube-api-access-vjwfg") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "kube-api-access-vjwfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:20:23.771453 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.771416 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:20:23.771453 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.771436 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:20:23.771741 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.771710 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f30a118-1479-4863-8c65-93fc24343143-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f30a118-1479-4863-8c65-93fc24343143" (UID: "5f30a118-1479-4863-8c65-93fc24343143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:20:23.870172 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.870145 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.870172 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.870169 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjwfg\" (UniqueName: \"kubernetes.io/projected/5f30a118-1479-4863-8c65-93fc24343143-kube-api-access-vjwfg\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.870350 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.870178 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.870350 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.870186 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.870350 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.870194 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f30a118-1479-4863-8c65-93fc24343143-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.870350 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.870209 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f30a118-1479-4863-8c65-93fc24343143-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:23.870350 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:23.870221 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f30a118-1479-4863-8c65-93fc24343143-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:20:24.527049 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.527012 2568 generic.go:356] "Generic (PLEG): container finished" podID="5f30a118-1479-4863-8c65-93fc24343143" containerID="04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e" exitCode=0 Jul 06 13:20:24.527220 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.527062 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" Jul 06 13:20:24.527220 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.527065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" event={"ID":"5f30a118-1479-4863-8c65-93fc24343143","Type":"ContainerDied","Data":"04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e"} Jul 06 13:20:24.527220 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.527175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q" event={"ID":"5f30a118-1479-4863-8c65-93fc24343143","Type":"ContainerDied","Data":"b85240a70792d803b9d5a4d59458427f2d2113a831a0e8326daaf30154d004b6"} Jul 06 13:20:24.527220 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.527201 2568 scope.go:117] "RemoveContainer" containerID="04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e" Jul 06 13:20:24.537602 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.537430 2568 scope.go:117] "RemoveContainer" containerID="6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566" Jul 06 13:20:24.547885 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.547859 2568 scope.go:117] "RemoveContainer" containerID="04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e" Jul 06 13:20:24.548370 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:20:24.548199 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e\": container with ID starting with 04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e not found: ID does not exist" containerID="04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e" Jul 06 13:20:24.548370 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.548235 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e"} err="failed to get container status \"04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e\": rpc error: code = NotFound desc = could not find container \"04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e\": container with ID starting with 04da2348fa7a8269abffbce8a1fd77312327cf543dcb218c607a3ed4ed3bf40e not found: ID does not exist" Jul 06 13:20:24.548370 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.548260 2568 scope.go:117] "RemoveContainer" containerID="6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566" Jul 06 13:20:24.548705 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:20:24.548677 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566\": container with ID starting with 6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566 not found: ID does not exist" containerID="6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566" Jul 06 13:20:24.548777 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.548723 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566"} err="failed to get container status \"6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566\": rpc error: code = NotFound desc = could not find container \"6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566\": container with ID starting with 6138b39a4ed83b8bb9246489fcaa1c1937ed0b5b6caebf5b6b3386cad5681566 not found: ID does not exist" Jul 06 13:20:24.551387 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.551364 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q"] Jul 06 13:20:24.553737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.553717 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69a5e1cc-openshift-default-bfc8f8fc5-lx56q"] Jul 06 13:20:24.590797 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:20:24.590774 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f30a118-1479-4863-8c65-93fc24343143" path="/var/lib/kubelet/pods/5f30a118-1479-4863-8c65-93fc24343143/volumes" Jul 06 13:21:16.109329 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.109294 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg"] Jul 06 13:21:16.109747 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.109643 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" Jul 06 13:21:16.109747 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.109656 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" Jul 06 13:21:16.109747 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.109681 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" Jul 06 13:21:16.109747 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.109686 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" Jul 06 13:21:16.109747 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.109746 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" Jul 06 13:21:16.112739 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.112723 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.115887 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.115869 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8bf2fb-openshift-default-dockercfg-zbs4m\"" Jul 06 13:21:16.136477 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.136446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg"] Jul 06 13:21:16.190126 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-credential-socket\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190126 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190129 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-socket\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190337 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-envoy\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190337 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190161 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-token\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190337 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-certs\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190337 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istiod-ca-cert\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190337 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qw8l\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-kube-api-access-6qw8l\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190337 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-data\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.190337 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.190314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-podinfo\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291584 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291556 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-credential-socket\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291595 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-socket\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-envoy\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-token\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-certs\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291737 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istiod-ca-cert\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qw8l\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-kube-api-access-6qw8l\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.291796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-data\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.292172 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.291820 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-podinfo\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.292172 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.292029 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-credential-socket\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.292434 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.292399 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-socket\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.292551 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.292457 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-certs\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.292623 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.292601 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-data\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.292743 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.292721 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istiod-ca-cert\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.294833 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.294805 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-envoy\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.295074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.295054 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-podinfo\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.305361 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.305340 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-token\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.305495 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.305479 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qw8l\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-kube-api-access-6qw8l\") pod \"e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.422706 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.422604 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:16.587834 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:21:16.587798 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4151e1a5_e67d_4379_82da_bc8a3e35e1ad.slice/crio-71f312788afedc3984b19cee330995109a7585f322acbf089295ee01cf31166d WatchSource:0}: Error finding container 71f312788afedc3984b19cee330995109a7585f322acbf089295ee01cf31166d: Status 404 returned error can't find the container with id 71f312788afedc3984b19cee330995109a7585f322acbf089295ee01cf31166d Jul 06 13:21:16.590046 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.590011 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:16.590138 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.590082 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:16.590138 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.590113 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:16.591550 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.591527 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg"] Jul 06 13:21:16.802467 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.802366 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" event={"ID":"4151e1a5-e67d-4379-82da-bc8a3e35e1ad","Type":"ContainerStarted","Data":"4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e"} Jul 06 13:21:16.802467 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.802436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" event={"ID":"4151e1a5-e67d-4379-82da-bc8a3e35e1ad","Type":"ContainerStarted","Data":"71f312788afedc3984b19cee330995109a7585f322acbf089295ee01cf31166d"} Jul 06 13:21:16.826519 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:16.826460 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" podStartSLOduration=0.826444267 podStartE2EDuration="826.444267ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:21:16.823689701 +0000 UTC m=+2060.764530562" watchObservedRunningTime="2026-07-06 13:21:16.826444267 +0000 UTC m=+2060.767285125" Jul 06 13:21:17.423601 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:17.423564 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:17.428696 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:17.428674 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:17.807992 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:17.807875 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:17.808949 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:17.808928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:24.222553 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.222522 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w"] Jul 06 13:21:24.223037 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.223019 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f30a118-1479-4863-8c65-93fc24343143" containerName="istio-proxy" Jul 06 13:21:24.226220 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.226202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.234018 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.234001 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cb2d4a-openshift-default-dockercfg-qhzmw\"" Jul 06 13:21:24.253806 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.253784 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w"] Jul 06 13:21:24.265300 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-envoy\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265415 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-token\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265415 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-credential-socket\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265506 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-certs\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265506 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265442 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-data\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265587 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbn55\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-kube-api-access-gbn55\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265587 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-socket\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265609 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-podinfo\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.265713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.265680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istiod-ca-cert\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366483 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-credential-socket\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-certs\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-data\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366525 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbn55\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-kube-api-access-gbn55\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-socket\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-podinfo\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istiod-ca-cert\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.366713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366686 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-envoy\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.367096 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-token\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.367096 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.366991 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-credential-socket\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.367096 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.367018 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-data\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.367212 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.367119 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-certs\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.367212 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.367160 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-socket\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.367502 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.367475 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istiod-ca-cert\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.369244 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.369226 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-podinfo\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.369316 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.369255 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-envoy\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.385964 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.385941 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbn55\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-kube-api-access-gbn55\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.404157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.404135 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-token\") pod \"e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.537576 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.537483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:24.696315 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.696275 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w"] Jul 06 13:21:24.696480 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:21:24.696442 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ec0b45_72b0_446e_b2ac_fe8c1c0292e8.slice/crio-dc401abc8acc8d2c6331e1bc0f0e5595989dd3a7d0dd4456e75554bc82e7e1fb WatchSource:0}: Error finding container dc401abc8acc8d2c6331e1bc0f0e5595989dd3a7d0dd4456e75554bc82e7e1fb: Status 404 returned error can't find the container with id dc401abc8acc8d2c6331e1bc0f0e5595989dd3a7d0dd4456e75554bc82e7e1fb Jul 06 13:21:24.699227 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.699173 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:24.699317 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.699266 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:24.699317 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.699301 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:24.846241 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.846159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" event={"ID":"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8","Type":"ContainerStarted","Data":"5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3"} Jul 06 13:21:24.846241 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.846207 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" event={"ID":"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8","Type":"ContainerStarted","Data":"dc401abc8acc8d2c6331e1bc0f0e5595989dd3a7d0dd4456e75554bc82e7e1fb"} Jul 06 13:21:24.890733 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:24.890670 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" podStartSLOduration=0.890650311 podStartE2EDuration="890.650311ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:21:24.887158654 +0000 UTC m=+2068.827999534" watchObservedRunningTime="2026-07-06 13:21:24.890650311 +0000 UTC m=+2068.831491164" Jul 06 13:21:25.538090 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:25.538058 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:25.543285 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:25.543263 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:25.851765 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:25.851733 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:25.852578 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:25.852557 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:32.809255 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.809218 2568 patch_prober.go:28] interesting pod/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 13:21:32.809759 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.809282 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 13:21:32.898818 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.898783 2568 generic.go:356] "Generic (PLEG): container finished" podID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerID="4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e" exitCode=0 Jul 06 13:21:32.899014 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.898837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" event={"ID":"4151e1a5-e67d-4379-82da-bc8a3e35e1ad","Type":"ContainerDied","Data":"4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e"} Jul 06 13:21:32.899309 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.899292 2568 scope.go:117] "RemoveContainer" containerID="4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e" Jul 06 13:21:32.900361 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.900331 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:32.900420 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.900399 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:32.900461 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:32.900438 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:33.909742 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:33.909702 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" event={"ID":"4151e1a5-e67d-4379-82da-bc8a3e35e1ad","Type":"ContainerStarted","Data":"a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833"} Jul 06 13:21:34.423602 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:34.423566 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:34.423841 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:34.423618 2568 patch_prober.go:28] interesting pod/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 13:21:34.423841 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:34.423715 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 13:21:35.424043 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:35.424006 2568 patch_prober.go:28] interesting pod/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 13:21:35.424444 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:35.424064 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 13:21:36.423711 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:36.423672 2568 patch_prober.go:28] interesting pod/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 13:21:36.423880 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:36.423732 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 13:21:37.423456 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:37.423409 2568 patch_prober.go:28] interesting pod/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 13:21:37.423881 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:37.423482 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 13:21:38.426595 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:38.426557 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:38.427069 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:38.426841 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:38.427617 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:38.427603 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:21:39.945730 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:39.945700 2568 generic.go:356] "Generic (PLEG): container finished" podID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerID="5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3" exitCode=0 Jul 06 13:21:39.946150 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:39.945771 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" event={"ID":"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8","Type":"ContainerDied","Data":"5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3"} Jul 06 13:21:39.946270 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:39.946256 2568 scope.go:117] "RemoveContainer" containerID="5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3" Jul 06 13:21:39.947289 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:39.947262 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:39.947348 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:39.947333 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:39.947394 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:39.947368 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:21:40.852397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:40.852367 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:40.952811 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:40.952771 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" event={"ID":"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8","Type":"ContainerStarted","Data":"62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b"} Jul 06 13:21:41.538539 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:41.538506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:41.539476 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:41.539455 2568 patch_prober.go:28] interesting pod/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 13:21:41.539529 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:41.539503 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 13:21:42.538837 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:42.538804 2568 patch_prober.go:28] interesting pod/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 13:21:42.539234 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:42.538864 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 13:21:43.538518 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:43.538486 2568 patch_prober.go:28] interesting pod/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 13:21:43.538730 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:43.538538 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 13:21:44.538422 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:44.538386 2568 patch_prober.go:28] interesting pod/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 13:21:44.538833 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:44.538443 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 13:21:45.542448 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:45.542413 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:45.542909 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:45.542691 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:45.543415 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:45.543400 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:21:56.701805 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:56.701781 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:21:56.709606 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:21:56.709586 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:23:26.991740 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:26.991709 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg"] Jul 06 13:23:26.992243 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:26.991925 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" containerID="cri-o://a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833" gracePeriod=30 Jul 06 13:23:32.251304 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.251246 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:23:32.354884 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.354848 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-socket\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.354904 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-token\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.354938 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-envoy\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.354976 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qw8l\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-kube-api-access-6qw8l\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355009 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istiod-ca-cert\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355305 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355080 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-data\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355305 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355113 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-certs\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355305 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355140 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-podinfo\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355305 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355163 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-credential-socket\") pod \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\" (UID: \"4151e1a5-e67d-4379-82da-bc8a3e35e1ad\") " Jul 06 13:23:32.355305 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355175 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:32.355557 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355444 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.355731 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355694 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:32.355731 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355712 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:23:32.355935 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.355895 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:32.356364 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.356337 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:32.357713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.357684 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-kube-api-access-6qw8l" (OuterVolumeSpecName: "kube-api-access-6qw8l") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "kube-api-access-6qw8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:23:32.357713 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.357695 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:23:32.358244 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.358215 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:32.358244 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.358223 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4151e1a5-e67d-4379-82da-bc8a3e35e1ad" (UID: "4151e1a5-e67d-4379-82da-bc8a3e35e1ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:23:32.456585 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456558 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.456585 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456587 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.456793 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456600 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qw8l\" (UniqueName: \"kubernetes.io/projected/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-kube-api-access-6qw8l\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.456793 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456613 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.456793 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456626 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.456793 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456653 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.456793 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456663 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.456793 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.456675 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4151e1a5-e67d-4379-82da-bc8a3e35e1ad-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:32.568694 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.568603 2568 generic.go:356] "Generic (PLEG): container finished" podID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerID="a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833" exitCode=0 Jul 06 13:23:32.568694 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.568683 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" Jul 06 13:23:32.568694 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.568685 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" event={"ID":"4151e1a5-e67d-4379-82da-bc8a3e35e1ad","Type":"ContainerDied","Data":"a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833"} Jul 06 13:23:32.568891 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.568776 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg" event={"ID":"4151e1a5-e67d-4379-82da-bc8a3e35e1ad","Type":"ContainerDied","Data":"71f312788afedc3984b19cee330995109a7585f322acbf089295ee01cf31166d"} Jul 06 13:23:32.568891 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.568800 2568 scope.go:117] "RemoveContainer" containerID="a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833" Jul 06 13:23:32.578696 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.578677 2568 scope.go:117] "RemoveContainer" containerID="4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e" Jul 06 13:23:32.586727 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.586707 2568 scope.go:117] "RemoveContainer" containerID="a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833" Jul 06 13:23:32.586995 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:23:32.586975 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833\": container with ID starting with a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833 not found: ID does not exist" containerID="a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833" Jul 06 13:23:32.587061 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.587008 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833"} err="failed to get container status \"a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833\": rpc error: code = NotFound desc = could not find container \"a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833\": container with ID starting with a23f86f8a0ebcd2be8b15bf02da70256e11d273b78eea32559cb0addc4508833 not found: ID does not exist" Jul 06 13:23:32.587061 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.587025 2568 scope.go:117] "RemoveContainer" containerID="4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e" Jul 06 13:23:32.587252 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:23:32.587236 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e\": container with ID starting with 4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e not found: ID does not exist" containerID="4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e" Jul 06 13:23:32.587310 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.587255 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e"} err="failed to get container status \"4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e\": rpc error: code = NotFound desc = could not find container \"4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e\": container with ID starting with 4e44c5afcd25efe3716e4914b6e338a24c32bed01f4ca23a9c219f046d1e510e not found: ID does not exist" Jul 06 13:23:32.592698 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.592673 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg"] Jul 06 13:23:32.599866 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:32.599836 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8bf2fb-openshift-default-868f947577-t4ccg"] Jul 06 13:23:34.590784 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:34.590752 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" path="/var/lib/kubelet/pods/4151e1a5-e67d-4379-82da-bc8a3e35e1ad/volumes" Jul 06 13:23:35.070913 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:35.070881 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w"] Jul 06 13:23:35.071145 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:35.071122 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" containerID="cri-o://62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b" gracePeriod=30 Jul 06 13:23:36.195705 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.195672 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92"] Jul 06 13:23:36.196089 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.196079 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" Jul 06 13:23:36.196132 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.196091 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" Jul 06 13:23:36.196132 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.196100 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" Jul 06 13:23:36.196132 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.196106 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" Jul 06 13:23:36.196225 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.196163 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" Jul 06 13:23:36.196225 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.196174 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4151e1a5-e67d-4379-82da-bc8a3e35e1ad" containerName="istio-proxy" Jul 06 13:23:36.200774 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.200758 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.204417 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.204387 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fdcd8d-openshift-default-dockercfg-b6ffg\"" Jul 06 13:23:36.218896 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.218868 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92"] Jul 06 13:23:36.289715 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.289679 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-certs\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.289715 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.289715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-data\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.289921 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.289733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-envoy\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.289921 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.289811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchxp\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-kube-api-access-tchxp\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.289921 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.289847 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-token\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.289921 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.289875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-socket\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.289921 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.289904 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-podinfo\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.290078 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.290005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-credential-socket\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.290113 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.290074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istiod-ca-cert\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.390948 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.390915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istiod-ca-cert\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.390948 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.390948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-certs\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391178 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.390965 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-data\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391178 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.390982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-envoy\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391178 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchxp\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-kube-api-access-tchxp\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391178 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-token\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391178 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-socket\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391178 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391128 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-podinfo\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391178 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-credential-socket\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391709 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391365 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-data\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391709 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391440 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-certs\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391828 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391763 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istiod-ca-cert\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391828 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391787 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-socket\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.391828 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.391818 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-credential-socket\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.393567 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.393547 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-envoy\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.393947 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.393926 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-podinfo\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.407722 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.407700 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchxp\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-kube-api-access-tchxp\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.407800 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.407766 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-token\") pod \"e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.511858 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.511775 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:36.669035 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.669012 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92"] Jul 06 13:23:36.670957 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:23:36.670925 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f57b37a_8c52_46e4_a30c_e85b3a7e6ee9.slice/crio-39b96ec5e2cf81c90d70d9500f77fe7dbc7388ab40d20915a583bef9fbbf801b WatchSource:0}: Error finding container 39b96ec5e2cf81c90d70d9500f77fe7dbc7388ab40d20915a583bef9fbbf801b: Status 404 returned error can't find the container with id 39b96ec5e2cf81c90d70d9500f77fe7dbc7388ab40d20915a583bef9fbbf801b Jul 06 13:23:36.673035 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.673003 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:23:36.673120 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.673065 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:23:36.673120 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:36.673095 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:23:37.598197 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:37.598164 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" event={"ID":"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9","Type":"ContainerStarted","Data":"10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8"} Jul 06 13:23:37.598599 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:37.598214 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" event={"ID":"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9","Type":"ContainerStarted","Data":"39b96ec5e2cf81c90d70d9500f77fe7dbc7388ab40d20915a583bef9fbbf801b"} Jul 06 13:23:37.626923 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:37.626872 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" podStartSLOduration=1.626855952 podStartE2EDuration="1.626855952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:23:37.624302381 +0000 UTC m=+2201.565143243" watchObservedRunningTime="2026-07-06 13:23:37.626855952 +0000 UTC m=+2201.567696875" Jul 06 13:23:38.512391 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:38.512356 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:38.517083 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:38.517058 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:38.602951 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:38.602922 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:38.603966 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:38.603946 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:40.323050 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.322605 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:23:40.424641 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424611 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-credential-socket\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.424825 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424654 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-socket\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.424825 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424680 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-podinfo\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.424825 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424701 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-envoy\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.424825 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424733 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbn55\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-kube-api-access-gbn55\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.424825 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424753 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-data\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.424825 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424767 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istiod-ca-cert\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.425148 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424830 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-token\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.425148 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424867 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-certs\") pod \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\" (UID: \"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8\") " Jul 06 13:23:40.425148 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424945 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:40.425148 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.424957 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:40.425323 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.425203 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:40.425323 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.425228 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.425323 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.425249 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.425323 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.425298 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:23:40.425975 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.425937 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:40.427148 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.427127 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-kube-api-access-gbn55" (OuterVolumeSpecName: "kube-api-access-gbn55") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "kube-api-access-gbn55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:23:40.427271 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.427250 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:23:40.427543 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.427525 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:23:40.427582 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.427552 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" (UID: "c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:23:40.526015 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.525986 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.526015 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.526013 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.526181 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.526026 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbn55\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-kube-api-access-gbn55\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.526181 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.526039 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.526181 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.526051 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.526181 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.526062 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.526181 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.526073 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:23:40.615576 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.615548 2568 generic.go:356] "Generic (PLEG): container finished" podID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerID="62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b" exitCode=0 Jul 06 13:23:40.615768 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.615605 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" Jul 06 13:23:40.615768 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.615654 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" event={"ID":"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8","Type":"ContainerDied","Data":"62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b"} Jul 06 13:23:40.615768 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.615710 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w" event={"ID":"c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8","Type":"ContainerDied","Data":"dc401abc8acc8d2c6331e1bc0f0e5595989dd3a7d0dd4456e75554bc82e7e1fb"} Jul 06 13:23:40.615768 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.615732 2568 scope.go:117] "RemoveContainer" containerID="62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b" Jul 06 13:23:40.625224 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.625207 2568 scope.go:117] "RemoveContainer" containerID="5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3" Jul 06 13:23:40.636265 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.636247 2568 scope.go:117] "RemoveContainer" containerID="62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b" Jul 06 13:23:40.636753 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:23:40.636729 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b\": container with ID starting with 62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b not found: ID does not exist" containerID="62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b" Jul 06 13:23:40.636838 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.636763 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b"} err="failed to get container status \"62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b\": rpc error: code = NotFound desc = could not find container \"62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b\": container with ID starting with 62f03e1cb8d08709b7308c1449a14fba725e7430df42f7ac1fdd1117c714bb8b not found: ID does not exist" Jul 06 13:23:40.636838 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.636785 2568 scope.go:117] "RemoveContainer" containerID="5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3" Jul 06 13:23:40.637076 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:23:40.637057 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3\": container with ID starting with 5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3 not found: ID does not exist" containerID="5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3" Jul 06 13:23:40.637126 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.637083 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3"} err="failed to get container status \"5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3\": rpc error: code = NotFound desc = could not find container \"5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3\": container with ID starting with 5fa3fb1f209db923ef22fbde1a0abb8ff7a0619a2052fa9049273ef4ad4df1e3 not found: ID does not exist" Jul 06 13:23:40.638593 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.638575 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w"] Jul 06 13:23:40.647370 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:40.647349 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cb2d4a-openshift-default-8cd5f9575-wgz7w"] Jul 06 13:23:42.592005 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:42.591972 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" path="/var/lib/kubelet/pods/c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8/volumes" Jul 06 13:23:51.678957 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:51.678928 2568 generic.go:356] "Generic (PLEG): container finished" podID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerID="10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8" exitCode=0 Jul 06 13:23:51.679480 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:51.679011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" event={"ID":"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9","Type":"ContainerDied","Data":"10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8"} Jul 06 13:23:51.679480 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:51.679440 2568 scope.go:117] "RemoveContainer" containerID="10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8" Jul 06 13:23:51.680483 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:51.680454 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:23:51.681547 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:51.681081 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:23:51.681547 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:51.681219 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:23:52.686896 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:52.686862 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" event={"ID":"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9","Type":"ContainerStarted","Data":"9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd"} Jul 06 13:23:53.512502 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:53.512461 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:53.513618 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:53.513593 2568 patch_prober.go:28] interesting pod/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 13:23:53.513771 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:53.513659 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 13:23:53.603796 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:53.603768 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:54.512518 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:54.512482 2568 patch_prober.go:28] interesting pod/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 13:23:54.512964 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:54.512537 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 13:23:55.512672 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:55.512611 2568 patch_prober.go:28] interesting pod/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 13:23:55.513083 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:55.512702 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 13:23:56.512837 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:56.512794 2568 patch_prober.go:28] interesting pod/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 13:23:56.513293 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:56.512852 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 13:23:57.516650 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:57.516602 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:23:57.517783 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:23:57.517761 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:24:12.313268 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:12.313230 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92"] Jul 06 13:24:12.313806 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:12.313482 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" containerID="cri-o://9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd" gracePeriod=30 Jul 06 13:24:12.519392 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:12.519355 2568 patch_prober.go:28] interesting pod/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 13:24:12.519558 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:12.519410 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 13:24:17.578557 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.578493 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:24:17.672722 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672688 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-socket\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.672906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672745 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-data\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.672906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672771 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tchxp\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-kube-api-access-tchxp\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.672906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672792 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-podinfo\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.672906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672823 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-token\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.672906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672848 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-credential-socket\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.672906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672872 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istiod-ca-cert\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.672906 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672907 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-envoy\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.673245 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.672941 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-certs\") pod \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\" (UID: \"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9\") " Jul 06 13:24:17.673245 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.673035 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:24:17.673245 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.673145 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:24:17.673423 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.673343 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.673423 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.673362 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.673737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.673695 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:24:17.673737 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.673698 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:24:17.673911 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.673842 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:24:17.675259 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.675230 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:24:17.675469 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.675448 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-kube-api-access-tchxp" (OuterVolumeSpecName: "kube-api-access-tchxp") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "kube-api-access-tchxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:24:17.675537 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.675472 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:24:17.675775 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.675756 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" (UID: "5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:24:17.774884 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.774850 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.774884 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.774879 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchxp\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-kube-api-access-tchxp\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.775074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.774893 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.775074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.774905 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.775074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.774919 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.775074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.774929 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.775074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.774939 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:24:17.820726 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.820692 2568 generic.go:356] "Generic (PLEG): container finished" podID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerID="9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd" exitCode=0 Jul 06 13:24:17.820858 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.820747 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" Jul 06 13:24:17.820858 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.820784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" event={"ID":"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9","Type":"ContainerDied","Data":"9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd"} Jul 06 13:24:17.820858 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.820834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92" event={"ID":"5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9","Type":"ContainerDied","Data":"39b96ec5e2cf81c90d70d9500f77fe7dbc7388ab40d20915a583bef9fbbf801b"} Jul 06 13:24:17.820858 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.820857 2568 scope.go:117] "RemoveContainer" containerID="9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd" Jul 06 13:24:17.830455 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.830437 2568 scope.go:117] "RemoveContainer" containerID="10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8" Jul 06 13:24:17.838533 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.838517 2568 scope.go:117] "RemoveContainer" containerID="9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd" Jul 06 13:24:17.838797 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:24:17.838776 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd\": container with ID starting with 9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd not found: ID does not exist" containerID="9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd" Jul 06 13:24:17.838876 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.838809 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd"} err="failed to get container status \"9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd\": rpc error: code = NotFound desc = could not find container \"9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd\": container with ID starting with 9c31ea6a69fe9fda16389e4c41d3be5c70c269e1eafe8defe819db01178ea9cd not found: ID does not exist" Jul 06 13:24:17.838876 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.838832 2568 scope.go:117] "RemoveContainer" containerID="10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8" Jul 06 13:24:17.839080 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:24:17.839065 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8\": container with ID starting with 10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8 not found: ID does not exist" containerID="10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8" Jul 06 13:24:17.839123 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.839086 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8"} err="failed to get container status \"10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8\": rpc error: code = NotFound desc = could not find container \"10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8\": container with ID starting with 10288600d05db0381bb351f1f4a765c550c9cc0866c5460675a82399afb926e8 not found: ID does not exist" Jul 06 13:24:17.850335 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.850315 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92"] Jul 06 13:24:17.860041 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:17.860019 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fdcd8d-openshift-default-75d9f8c745-r6z92"] Jul 06 13:24:18.592287 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:18.592254 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" path="/var/lib/kubelet/pods/5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9/volumes" Jul 06 13:24:21.733017 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.732985 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782"] Jul 06 13:24:21.733386 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733335 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" Jul 06 13:24:21.733386 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733346 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" Jul 06 13:24:21.733386 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733354 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" Jul 06 13:24:21.733386 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733360 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" Jul 06 13:24:21.733386 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733379 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" Jul 06 13:24:21.733386 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733384 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" Jul 06 13:24:21.733579 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733434 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" Jul 06 13:24:21.733579 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733445 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" Jul 06 13:24:21.733579 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.733451 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec0b45-72b0-446e-b2ac-fe8c1c0292e8" containerName="istio-proxy" Jul 06 13:24:21.737866 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.737846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.743211 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.743188 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-164b27-openshift-default-dockercfg-qgd65\"" Jul 06 13:24:21.787161 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.787129 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782"] Jul 06 13:24:21.808026 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.807983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28620c-289d-4bd3-ad35-a96059384d6a-istiod-ca-cert\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808154 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808028 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ksd\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-kube-api-access-h7ksd\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808154 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-certs\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808154 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-data\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808259 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808173 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-socket\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808259 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-envoy\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808259 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808238 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-token\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808347 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-podinfo\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.808347 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.808309 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-credential-socket\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.909444 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28620c-289d-4bd3-ad35-a96059384d6a-istiod-ca-cert\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.909444 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ksd\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-kube-api-access-h7ksd\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.909702 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-certs\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.909745 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-data\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.909785 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-socket\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.909926 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-envoy\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.909980 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909939 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-token\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.910032 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.909977 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-podinfo\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.910089 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.910032 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-credential-socket\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.910089 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.910067 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-certs\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.910192 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.910094 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-socket\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.910192 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.910133 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-data\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.910288 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.910219 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28620c-289d-4bd3-ad35-a96059384d6a-istiod-ca-cert\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.910365 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.910340 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-credential-socket\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.912328 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.912307 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-envoy\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.912446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.912429 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-podinfo\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.922709 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.922686 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-token\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:21.922842 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:21.922823 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ksd\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-kube-api-access-h7ksd\") pod \"e2e-shared-a-164b27-openshift-default-5668bb654c-w9782\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:22.048331 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.048248 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:22.200029 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.199998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782"] Jul 06 13:24:22.201652 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:24:22.201603 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f28620c_289d_4bd3_ad35_a96059384d6a.slice/crio-fdc22935036b2065adeeb349bb97091f3722d2737f5de53fab2c15ec423aadab WatchSource:0}: Error finding container fdc22935036b2065adeeb349bb97091f3722d2737f5de53fab2c15ec423aadab: Status 404 returned error can't find the container with id fdc22935036b2065adeeb349bb97091f3722d2737f5de53fab2c15ec423aadab Jul 06 13:24:22.203908 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.203876 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:24:22.203987 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.203950 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:24:22.203987 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.203983 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:24:22.850929 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.850890 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" event={"ID":"3f28620c-289d-4bd3-ad35-a96059384d6a","Type":"ContainerStarted","Data":"79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145"} Jul 06 13:24:22.851354 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.850950 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" event={"ID":"3f28620c-289d-4bd3-ad35-a96059384d6a","Type":"ContainerStarted","Data":"fdc22935036b2065adeeb349bb97091f3722d2737f5de53fab2c15ec423aadab"} Jul 06 13:24:22.890036 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:22.889961 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" podStartSLOduration=1.889944976 podStartE2EDuration="1.889944976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:24:22.889576453 +0000 UTC m=+2246.830417313" watchObservedRunningTime="2026-07-06 13:24:22.889944976 +0000 UTC m=+2246.830785834" Jul 06 13:24:23.048997 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:23.048921 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:23.053854 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:23.053831 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:23.857181 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:23.857140 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:23.858278 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:23.858254 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:36.938919 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:36.938885 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerID="79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145" exitCode=0 Jul 06 13:24:36.939339 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:36.938961 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" event={"ID":"3f28620c-289d-4bd3-ad35-a96059384d6a","Type":"ContainerDied","Data":"79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145"} Jul 06 13:24:36.939396 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:36.939352 2568 scope.go:117] "RemoveContainer" containerID="79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145" Jul 06 13:24:36.940596 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:36.940558 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:24:36.940704 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:36.940683 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:24:36.940759 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:36.940743 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 13:24:37.945741 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:37.945700 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" event={"ID":"3f28620c-289d-4bd3-ad35-a96059384d6a","Type":"ContainerStarted","Data":"432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79"} Jul 06 13:24:38.048685 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:38.048620 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:38.049670 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:38.049648 2568 patch_prober.go:28] interesting pod/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 13:24:38.049743 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:38.049699 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 13:24:38.858073 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:38.858031 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:39.048914 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:39.048878 2568 patch_prober.go:28] interesting pod/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 13:24:39.049294 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:39.048933 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 13:24:40.049062 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:40.049026 2568 patch_prober.go:28] interesting pod/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 13:24:40.049455 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:40.049075 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 13:24:41.049332 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:41.049291 2568 patch_prober.go:28] interesting pod/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 13:24:41.049844 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:41.049352 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 13:24:42.048876 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:42.048839 2568 patch_prober.go:28] interesting pod/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 13:24:42.049081 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:42.048903 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 13:24:43.053230 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:43.053198 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:24:43.054182 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:24:43.054160 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:25:18.710852 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.710821 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq"] Jul 06 13:25:18.711279 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.711162 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" Jul 06 13:25:18.711279 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.711172 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" Jul 06 13:25:18.711279 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.711248 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f57b37a-8c52-46e4-a30c-e85b3a7e6ee9" containerName="istio-proxy" Jul 06 13:25:18.714340 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.714323 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.719764 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.719738 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"kube-root-ca.crt\"" Jul 06 13:25:18.720771 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.720748 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"default-dockercfg-xhkc8\"" Jul 06 13:25:18.720911 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.720773 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"openshift-service-ca.crt\"" Jul 06 13:25:18.720911 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.720773 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"auth-test-model-03b528-kserve-self-signed-certs\"" Jul 06 13:25:18.736109 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.736078 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq"] Jul 06 13:25:18.856275 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.856239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tls-certs\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.856470 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.856298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tmp-dir\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.856470 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.856320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-dshm\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.856470 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.856383 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-home\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.856470 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.856407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-model-cache\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.856470 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.856437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8znd\" (UniqueName: \"kubernetes.io/projected/7c3a1113-ee55-4795-ac27-79d9d7568c5f-kube-api-access-f8znd\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.957745 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.957707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tmp-dir\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.957952 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.957750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-dshm\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.957952 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.957784 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-home\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.957952 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.957803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-model-cache\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.957952 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.957828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8znd\" (UniqueName: \"kubernetes.io/projected/7c3a1113-ee55-4795-ac27-79d9d7568c5f-kube-api-access-f8znd\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.958173 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.957975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tls-certs\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.958224 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.958167 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tmp-dir\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.958224 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.958201 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-home\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.958333 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.958310 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-model-cache\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.960319 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.960295 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-dshm\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.960588 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.960566 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tls-certs\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:18.969111 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:18.969042 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8znd\" (UniqueName: \"kubernetes.io/projected/7c3a1113-ee55-4795-ac27-79d9d7568c5f-kube-api-access-f8znd\") pod \"auth-test-model-03b528-kserve-65cc56f95f-md7hq\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:19.023775 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:19.023732 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:19.166995 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:19.166968 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq"] Jul 06 13:25:19.169538 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:25:19.169512 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c3a1113_ee55_4795_ac27_79d9d7568c5f.slice/crio-33820bf0278977c5a52c3438f0d903fd3891a45f29a921ff6d9e06b279754dc5 WatchSource:0}: Error finding container 33820bf0278977c5a52c3438f0d903fd3891a45f29a921ff6d9e06b279754dc5: Status 404 returned error can't find the container with id 33820bf0278977c5a52c3438f0d903fd3891a45f29a921ff6d9e06b279754dc5 Jul 06 13:25:19.189028 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:19.188999 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" event={"ID":"7c3a1113-ee55-4795-ac27-79d9d7568c5f","Type":"ContainerStarted","Data":"33820bf0278977c5a52c3438f0d903fd3891a45f29a921ff6d9e06b279754dc5"} Jul 06 13:25:20.196436 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:20.196393 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" event={"ID":"7c3a1113-ee55-4795-ac27-79d9d7568c5f","Type":"ContainerStarted","Data":"634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d"} Jul 06 13:25:20.227529 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:20.227479 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" podStartSLOduration=2.227464427 podStartE2EDuration="2.227464427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:25:20.224817761 +0000 UTC m=+2304.165658621" watchObservedRunningTime="2026-07-06 13:25:20.227464427 +0000 UTC m=+2304.168305310" Jul 06 13:25:29.024237 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:29.024187 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:29.024626 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:29.024263 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:29.036895 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:29.036871 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:25:29.254706 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:25:29.254681 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:26:56.739202 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:26:56.739175 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:26:56.747450 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:26:56.747425 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8cmpd_f46d4222-907b-4cb6-8217-38e8228ee9d6/ovn-acl-logging/0.log" Jul 06 13:27:19.291282 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:19.291205 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq"] Jul 06 13:27:28.596103 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.596067 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j"] Jul 06 13:27:28.599836 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.599814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.609473 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.609446 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-164b27\"/\"openshift-service-ca.crt\"" Jul 06 13:27:28.609599 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.609497 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-164b27\"/\"default-dockercfg-fm54g\"" Jul 06 13:27:28.609599 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.609525 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-164b27\"/\"sub-test-model-164b27-kserve-self-signed-certs\"" Jul 06 13:27:28.609599 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.609584 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-164b27\"/\"kube-root-ca.crt\"" Jul 06 13:27:28.642394 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.642372 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j"] Jul 06 13:27:28.754381 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.754346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-dshm\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.754577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.754404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dda7cca4-af08-49f3-885f-bf505aa71413-tls-certs\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.754577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.754486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27fk\" (UniqueName: \"kubernetes.io/projected/dda7cca4-af08-49f3-885f-bf505aa71413-kube-api-access-x27fk\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.754577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.754522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-home\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.754577 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.754562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-tmp-dir\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.754767 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.754617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-model-cache\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.855593 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.855564 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-model-cache\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.855787 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.855601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-dshm\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.855787 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.855662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dda7cca4-af08-49f3-885f-bf505aa71413-tls-certs\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.855787 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.855703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27fk\" (UniqueName: \"kubernetes.io/projected/dda7cca4-af08-49f3-885f-bf505aa71413-kube-api-access-x27fk\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.855787 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.855733 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-home\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.855787 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.855769 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-tmp-dir\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.856138 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.856046 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-model-cache\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.856202 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.856133 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-tmp-dir\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.856202 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.856191 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-home\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.858083 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.858059 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-dshm\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.858454 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.858434 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dda7cca4-af08-49f3-885f-bf505aa71413-tls-certs\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.871283 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.871259 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27fk\" (UniqueName: \"kubernetes.io/projected/dda7cca4-af08-49f3-885f-bf505aa71413-kube-api-access-x27fk\") pod \"sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:28.909499 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:28.909474 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:29.052773 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:29.052747 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j"] Jul 06 13:27:29.055109 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:27:29.055083 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddda7cca4_af08_49f3_885f_bf505aa71413.slice/crio-806818be483eb45fd1a7269980ece87f5533a5ccb3a22301c79b5a049fa88ee0 WatchSource:0}: Error finding container 806818be483eb45fd1a7269980ece87f5533a5ccb3a22301c79b5a049fa88ee0: Status 404 returned error can't find the container with id 806818be483eb45fd1a7269980ece87f5533a5ccb3a22301c79b5a049fa88ee0 Jul 06 13:27:29.888446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:29.888410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" event={"ID":"dda7cca4-af08-49f3-885f-bf505aa71413","Type":"ContainerStarted","Data":"319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e"} Jul 06 13:27:29.888446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:29.888453 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" event={"ID":"dda7cca4-af08-49f3-885f-bf505aa71413","Type":"ContainerStarted","Data":"806818be483eb45fd1a7269980ece87f5533a5ccb3a22301c79b5a049fa88ee0"} Jul 06 13:27:29.923476 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:29.923428 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" podStartSLOduration=1.9234138509999998 podStartE2EDuration="1.923413851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:27:29.919299208 +0000 UTC m=+2433.860140067" watchObservedRunningTime="2026-07-06 13:27:29.923413851 +0000 UTC m=+2433.864254718" Jul 06 13:27:34.311826 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.311778 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" podUID="7c3a1113-ee55-4795-ac27-79d9d7568c5f" containerName="main" containerID="cri-o://634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d" gracePeriod=45 Jul 06 13:27:34.554311 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.554280 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:27:34.714839 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.714807 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tmp-dir\") pod \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " Jul 06 13:27:34.715019 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.714887 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-home\") pod \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " Jul 06 13:27:34.715019 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.714910 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-model-cache\") pod \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " Jul 06 13:27:34.715019 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.714931 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8znd\" (UniqueName: \"kubernetes.io/projected/7c3a1113-ee55-4795-ac27-79d9d7568c5f-kube-api-access-f8znd\") pod \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " Jul 06 13:27:34.715019 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.714965 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-dshm\") pod \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " Jul 06 13:27:34.715248 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.715025 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tls-certs\") pod \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\" (UID: \"7c3a1113-ee55-4795-ac27-79d9d7568c5f\") " Jul 06 13:27:34.715248 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.715139 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c3a1113-ee55-4795-ac27-79d9d7568c5f" (UID: "7c3a1113-ee55-4795-ac27-79d9d7568c5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:27:34.715248 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.715155 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-home" (OuterVolumeSpecName: "home") pod "7c3a1113-ee55-4795-ac27-79d9d7568c5f" (UID: "7c3a1113-ee55-4795-ac27-79d9d7568c5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:27:34.715248 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.715194 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c3a1113-ee55-4795-ac27-79d9d7568c5f" (UID: "7c3a1113-ee55-4795-ac27-79d9d7568c5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:27:34.715427 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.715355 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tmp-dir\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:27:34.715427 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.715375 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-home\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:27:34.715427 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.715389 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-model-cache\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:27:34.717454 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.717426 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-dshm" (OuterVolumeSpecName: "dshm") pod "7c3a1113-ee55-4795-ac27-79d9d7568c5f" (UID: "7c3a1113-ee55-4795-ac27-79d9d7568c5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:27:34.717572 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.717478 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c3a1113-ee55-4795-ac27-79d9d7568c5f" (UID: "7c3a1113-ee55-4795-ac27-79d9d7568c5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:27:34.717572 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.717493 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3a1113-ee55-4795-ac27-79d9d7568c5f-kube-api-access-f8znd" (OuterVolumeSpecName: "kube-api-access-f8znd") pod "7c3a1113-ee55-4795-ac27-79d9d7568c5f" (UID: "7c3a1113-ee55-4795-ac27-79d9d7568c5f"). InnerVolumeSpecName "kube-api-access-f8znd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:27:34.816702 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.816671 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3a1113-ee55-4795-ac27-79d9d7568c5f-dshm\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:27:34.816702 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.816696 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3a1113-ee55-4795-ac27-79d9d7568c5f-tls-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:27:34.816702 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.816707 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8znd\" (UniqueName: \"kubernetes.io/projected/7c3a1113-ee55-4795-ac27-79d9d7568c5f-kube-api-access-f8znd\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:27:34.916520 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.916486 2568 generic.go:356] "Generic (PLEG): container finished" podID="7c3a1113-ee55-4795-ac27-79d9d7568c5f" containerID="634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d" exitCode=0 Jul 06 13:27:34.916703 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.916533 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" Jul 06 13:27:34.916703 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.916544 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" event={"ID":"7c3a1113-ee55-4795-ac27-79d9d7568c5f","Type":"ContainerDied","Data":"634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d"} Jul 06 13:27:34.916703 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.916578 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq" event={"ID":"7c3a1113-ee55-4795-ac27-79d9d7568c5f","Type":"ContainerDied","Data":"33820bf0278977c5a52c3438f0d903fd3891a45f29a921ff6d9e06b279754dc5"} Jul 06 13:27:34.916703 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.916594 2568 scope.go:117] "RemoveContainer" containerID="634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d" Jul 06 13:27:34.926227 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.926203 2568 scope.go:117] "RemoveContainer" containerID="634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d" Jul 06 13:27:34.926493 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:27:34.926471 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d\": container with ID starting with 634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d not found: ID does not exist" containerID="634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d" Jul 06 13:27:34.926539 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.926503 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d"} err="failed to get container status \"634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d\": rpc error: code = NotFound desc = could not find container \"634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d\": container with ID starting with 634c9e6a1c80ea586c52d052bdc6e2bc277831b29ead94e97b7554c319ae093d not found: ID does not exist" Jul 06 13:27:34.959303 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.958315 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq"] Jul 06 13:27:34.965992 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:34.965918 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/auth-test-model-03b528-kserve-65cc56f95f-md7hq"] Jul 06 13:27:36.591651 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:36.591608 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3a1113-ee55-4795-ac27-79d9d7568c5f" path="/var/lib/kubelet/pods/7c3a1113-ee55-4795-ac27-79d9d7568c5f/volumes" Jul 06 13:27:38.910552 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:38.910513 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:38.911025 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:38.910571 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:38.922784 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:38.922760 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:27:38.949949 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:27:38.949930 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:28:24.203360 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:24.203329 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j"] Jul 06 13:28:39.223710 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.223659 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" podUID="dda7cca4-af08-49f3-885f-bf505aa71413" containerName="main" containerID="cri-o://319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e" gracePeriod=45 Jul 06 13:28:39.479250 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.479193 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:28:39.488799 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.488772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-dshm\") pod \"dda7cca4-af08-49f3-885f-bf505aa71413\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " Jul 06 13:28:39.488874 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.488802 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-tmp-dir\") pod \"dda7cca4-af08-49f3-885f-bf505aa71413\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " Jul 06 13:28:39.488874 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.488837 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x27fk\" (UniqueName: \"kubernetes.io/projected/dda7cca4-af08-49f3-885f-bf505aa71413-kube-api-access-x27fk\") pod \"dda7cca4-af08-49f3-885f-bf505aa71413\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " Jul 06 13:28:39.488971 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.488879 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-home\") pod \"dda7cca4-af08-49f3-885f-bf505aa71413\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " Jul 06 13:28:39.488971 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.488958 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-model-cache\") pod \"dda7cca4-af08-49f3-885f-bf505aa71413\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " Jul 06 13:28:39.489258 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.489006 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dda7cca4-af08-49f3-885f-bf505aa71413-tls-certs\") pod \"dda7cca4-af08-49f3-885f-bf505aa71413\" (UID: \"dda7cca4-af08-49f3-885f-bf505aa71413\") " Jul 06 13:28:39.489258 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.489052 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dda7cca4-af08-49f3-885f-bf505aa71413" (UID: "dda7cca4-af08-49f3-885f-bf505aa71413"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:28:39.489258 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.489167 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-home" (OuterVolumeSpecName: "home") pod "dda7cca4-af08-49f3-885f-bf505aa71413" (UID: "dda7cca4-af08-49f3-885f-bf505aa71413"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:28:39.489258 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.489236 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-model-cache" (OuterVolumeSpecName: "model-cache") pod "dda7cca4-af08-49f3-885f-bf505aa71413" (UID: "dda7cca4-af08-49f3-885f-bf505aa71413"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:28:39.489485 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.489319 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-tmp-dir\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:28:39.489485 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.489336 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-home\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:28:39.489485 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.489349 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-model-cache\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:28:39.491082 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.491056 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-dshm" (OuterVolumeSpecName: "dshm") pod "dda7cca4-af08-49f3-885f-bf505aa71413" (UID: "dda7cca4-af08-49f3-885f-bf505aa71413"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:28:39.491197 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.491149 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda7cca4-af08-49f3-885f-bf505aa71413-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dda7cca4-af08-49f3-885f-bf505aa71413" (UID: "dda7cca4-af08-49f3-885f-bf505aa71413"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:28:39.491257 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.491232 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda7cca4-af08-49f3-885f-bf505aa71413-kube-api-access-x27fk" (OuterVolumeSpecName: "kube-api-access-x27fk") pod "dda7cca4-af08-49f3-885f-bf505aa71413" (UID: "dda7cca4-af08-49f3-885f-bf505aa71413"). InnerVolumeSpecName "kube-api-access-x27fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:28:39.590267 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.590227 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dda7cca4-af08-49f3-885f-bf505aa71413-tls-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:28:39.590267 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.590262 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dda7cca4-af08-49f3-885f-bf505aa71413-dshm\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:28:39.590267 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:39.590272 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27fk\" (UniqueName: \"kubernetes.io/projected/dda7cca4-af08-49f3-885f-bf505aa71413-kube-api-access-x27fk\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:28:40.269784 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.269748 2568 generic.go:356] "Generic (PLEG): container finished" podID="dda7cca4-af08-49f3-885f-bf505aa71413" containerID="319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e" exitCode=0 Jul 06 13:28:40.270265 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.269792 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" Jul 06 13:28:40.270265 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.269834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" event={"ID":"dda7cca4-af08-49f3-885f-bf505aa71413","Type":"ContainerDied","Data":"319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e"} Jul 06 13:28:40.270265 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.269885 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j" event={"ID":"dda7cca4-af08-49f3-885f-bf505aa71413","Type":"ContainerDied","Data":"806818be483eb45fd1a7269980ece87f5533a5ccb3a22301c79b5a049fa88ee0"} Jul 06 13:28:40.270265 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.269909 2568 scope.go:117] "RemoveContainer" containerID="319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e" Jul 06 13:28:40.279530 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.279508 2568 scope.go:117] "RemoveContainer" containerID="319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e" Jul 06 13:28:40.279788 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:28:40.279768 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e\": container with ID starting with 319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e not found: ID does not exist" containerID="319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e" Jul 06 13:28:40.279872 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.279795 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e"} err="failed to get container status \"319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e\": rpc error: code = NotFound desc = could not find container \"319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e\": container with ID starting with 319ee114105a57e446e390d01c383bf455369d28f1a980976d13d300fc67bb1e not found: ID does not exist" Jul 06 13:28:40.296585 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.296552 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j"] Jul 06 13:28:40.301623 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.301598 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-164b27/sub-test-model-164b27-kserve-7bdfbc8f6c-55k9j"] Jul 06 13:28:40.592925 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:40.592835 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dda7cca4-af08-49f3-885f-bf505aa71413" path="/var/lib/kubelet/pods/dda7cca4-af08-49f3-885f-bf505aa71413/volumes" Jul 06 13:28:52.522543 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.522452 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn"] Jul 06 13:28:52.523149 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.522832 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3a1113-ee55-4795-ac27-79d9d7568c5f" containerName="main" Jul 06 13:28:52.523149 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.522844 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3a1113-ee55-4795-ac27-79d9d7568c5f" containerName="main" Jul 06 13:28:52.523149 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.522862 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda7cca4-af08-49f3-885f-bf505aa71413" containerName="main" Jul 06 13:28:52.523149 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.522868 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda7cca4-af08-49f3-885f-bf505aa71413" containerName="main" Jul 06 13:28:52.523149 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.522935 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dda7cca4-af08-49f3-885f-bf505aa71413" containerName="main" Jul 06 13:28:52.523149 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.522946 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3a1113-ee55-4795-ac27-79d9d7568c5f" containerName="main" Jul 06 13:28:52.527219 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.527196 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.530462 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.530445 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"kube-root-ca.crt\"" Jul 06 13:28:52.531500 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.531481 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"openshift-service-ca.crt\"" Jul 06 13:28:52.531580 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.531561 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"rate-test-model-03b528-kserve-self-signed-certs\"" Jul 06 13:28:52.531671 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.531655 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-03b528\"/\"default-dockercfg-xhkc8\"" Jul 06 13:28:52.538932 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.538902 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn"] Jul 06 13:28:52.608874 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.608846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-home\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.608874 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.608877 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-model-cache\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.609094 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.608901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxgc\" (UniqueName: \"kubernetes.io/projected/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-kube-api-access-4fxgc\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.609094 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.608976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tls-certs\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.609094 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.609022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tmp-dir\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.609094 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.609040 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-dshm\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710356 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-home\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710521 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-model-cache\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710521 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710396 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxgc\" (UniqueName: \"kubernetes.io/projected/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-kube-api-access-4fxgc\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710521 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tls-certs\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710521 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tmp-dir\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710774 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710536 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-dshm\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710836 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710785 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-home\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710836 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710818 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-model-cache\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.710967 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.710940 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tmp-dir\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.712976 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.712952 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-dshm\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.713217 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.713198 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tls-certs\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.721590 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.721563 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxgc\" (UniqueName: \"kubernetes.io/projected/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-kube-api-access-4fxgc\") pod \"rate-test-model-03b528-kserve-74b665bcd7-brxmn\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.836882 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.836787 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:28:52.981509 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:52.981483 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn"] Jul 06 13:28:52.983143 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:28:52.983108 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55e447da_fcdd_4dc6_94f7_db0b8d97f37d.slice/crio-e8dc76634204177d178f35daa71ceaffe14566f0b69dd7835ffb58f6fd7186d8 WatchSource:0}: Error finding container e8dc76634204177d178f35daa71ceaffe14566f0b69dd7835ffb58f6fd7186d8: Status 404 returned error can't find the container with id e8dc76634204177d178f35daa71ceaffe14566f0b69dd7835ffb58f6fd7186d8 Jul 06 13:28:53.343569 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:53.343532 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" event={"ID":"55e447da-fcdd-4dc6-94f7-db0b8d97f37d","Type":"ContainerStarted","Data":"d98ebedaebbd61cbc1f94b04451ac684b8caaa28cf26567b8e37c8bab5cc7f41"} Jul 06 13:28:53.343761 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:28:53.343577 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" event={"ID":"55e447da-fcdd-4dc6-94f7-db0b8d97f37d","Type":"ContainerStarted","Data":"e8dc76634204177d178f35daa71ceaffe14566f0b69dd7835ffb58f6fd7186d8"} Jul 06 13:29:02.837536 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:02.837497 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:29:02.838100 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:02.837552 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:29:02.850218 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:02.850184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:29:02.875151 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:02.875087 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" podStartSLOduration=10.875068788 podStartE2EDuration="10.875068788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:28:53.369559676 +0000 UTC m=+2517.310400536" watchObservedRunningTime="2026-07-06 13:29:02.875068788 +0000 UTC m=+2526.815909647" Jul 06 13:29:03.411181 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:03.411156 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:29:34.428262 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:34.428227 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn"] Jul 06 13:29:49.448459 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.448385 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" podUID="55e447da-fcdd-4dc6-94f7-db0b8d97f37d" containerName="main" containerID="cri-o://d98ebedaebbd61cbc1f94b04451ac684b8caaa28cf26567b8e37c8bab5cc7f41" gracePeriod=45 Jul 06 13:29:49.645974 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.645941 2568 generic.go:356] "Generic (PLEG): container finished" podID="55e447da-fcdd-4dc6-94f7-db0b8d97f37d" containerID="d98ebedaebbd61cbc1f94b04451ac684b8caaa28cf26567b8e37c8bab5cc7f41" exitCode=0 Jul 06 13:29:49.646132 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.646018 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" event={"ID":"55e447da-fcdd-4dc6-94f7-db0b8d97f37d","Type":"ContainerDied","Data":"d98ebedaebbd61cbc1f94b04451ac684b8caaa28cf26567b8e37c8bab5cc7f41"} Jul 06 13:29:49.700930 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.697850 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:29:49.828505 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828476 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tmp-dir\") pod \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " Jul 06 13:29:49.828705 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828522 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-model-cache\") pod \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " Jul 06 13:29:49.828705 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828539 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-dshm\") pod \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " Jul 06 13:29:49.828705 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828593 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-home\") pod \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " Jul 06 13:29:49.828705 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828617 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fxgc\" (UniqueName: \"kubernetes.io/projected/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-kube-api-access-4fxgc\") pod \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " Jul 06 13:29:49.828705 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828674 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tls-certs\") pod \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\" (UID: \"55e447da-fcdd-4dc6-94f7-db0b8d97f37d\") " Jul 06 13:29:49.828975 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828838 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55e447da-fcdd-4dc6-94f7-db0b8d97f37d" (UID: "55e447da-fcdd-4dc6-94f7-db0b8d97f37d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:29:49.828975 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828831 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-model-cache" (OuterVolumeSpecName: "model-cache") pod "55e447da-fcdd-4dc6-94f7-db0b8d97f37d" (UID: "55e447da-fcdd-4dc6-94f7-db0b8d97f37d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:29:49.828975 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.828910 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-home" (OuterVolumeSpecName: "home") pod "55e447da-fcdd-4dc6-94f7-db0b8d97f37d" (UID: "55e447da-fcdd-4dc6-94f7-db0b8d97f37d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:29:49.830878 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.830848 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-dshm" (OuterVolumeSpecName: "dshm") pod "55e447da-fcdd-4dc6-94f7-db0b8d97f37d" (UID: "55e447da-fcdd-4dc6-94f7-db0b8d97f37d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:29:49.831000 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.830953 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-kube-api-access-4fxgc" (OuterVolumeSpecName: "kube-api-access-4fxgc") pod "55e447da-fcdd-4dc6-94f7-db0b8d97f37d" (UID: "55e447da-fcdd-4dc6-94f7-db0b8d97f37d"). InnerVolumeSpecName "kube-api-access-4fxgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:29:49.831000 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.830950 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55e447da-fcdd-4dc6-94f7-db0b8d97f37d" (UID: "55e447da-fcdd-4dc6-94f7-db0b8d97f37d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:29:49.930213 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.930175 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tmp-dir\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:29:49.930213 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.930208 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-model-cache\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:29:49.930213 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.930217 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-dshm\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:29:49.930446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.930226 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-home\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:29:49.930446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.930235 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxgc\" (UniqueName: \"kubernetes.io/projected/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-kube-api-access-4fxgc\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:29:49.930446 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:49.930243 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e447da-fcdd-4dc6-94f7-db0b8d97f37d-tls-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:29:50.653534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:50.653504 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" Jul 06 13:29:50.654054 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:50.653526 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn" event={"ID":"55e447da-fcdd-4dc6-94f7-db0b8d97f37d","Type":"ContainerDied","Data":"e8dc76634204177d178f35daa71ceaffe14566f0b69dd7835ffb58f6fd7186d8"} Jul 06 13:29:50.654054 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:50.653591 2568 scope.go:117] "RemoveContainer" containerID="d98ebedaebbd61cbc1f94b04451ac684b8caaa28cf26567b8e37c8bab5cc7f41" Jul 06 13:29:50.673293 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:50.673265 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn"] Jul 06 13:29:50.678199 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:50.678175 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-03b528/rate-test-model-03b528-kserve-74b665bcd7-brxmn"] Jul 06 13:29:52.590703 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:29:52.590674 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e447da-fcdd-4dc6-94f7-db0b8d97f37d" path="/var/lib/kubelet/pods/55e447da-fcdd-4dc6-94f7-db0b8d97f37d/volumes" Jul 06 13:30:18.895370 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:18.895298 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782"] Jul 06 13:30:18.895848 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:18.895513 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" containerID="cri-o://432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79" gracePeriod=30 Jul 06 13:30:24.152946 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.152920 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:30:24.237952 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.237880 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-socket\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.237952 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.237940 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-podinfo\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.237961 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-envoy\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.237979 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-credential-socket\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238007 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28620c-289d-4bd3-ad35-a96059384d6a-istiod-ca-cert\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238044 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-token\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238074 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7ksd\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-kube-api-access-h7ksd\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238113 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-certs\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238157 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238146 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:30:24.238527 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238166 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-data\") pod \"3f28620c-289d-4bd3-ad35-a96059384d6a\" (UID: \"3f28620c-289d-4bd3-ad35-a96059384d6a\") " Jul 06 13:30:24.238527 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238440 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.238527 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238455 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:30:24.238527 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238476 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:30:24.238733 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.238537 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f28620c-289d-4bd3-ad35-a96059384d6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:30:24.239647 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.239607 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:30:24.240534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.240503 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-kube-api-access-h7ksd" (OuterVolumeSpecName: "kube-api-access-h7ksd") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "kube-api-access-h7ksd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:30:24.240534 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.240525 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:30:24.240818 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.240798 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:30:24.240818 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.240804 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f28620c-289d-4bd3-ad35-a96059384d6a" (UID: "3f28620c-289d-4bd3-ad35-a96059384d6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 13:30:24.339538 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339504 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-token\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.339538 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339532 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ksd\" (UniqueName: \"kubernetes.io/projected/3f28620c-289d-4bd3-ad35-a96059384d6a-kube-api-access-h7ksd\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.339538 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339542 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-workload-certs\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.339801 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339550 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-data\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.339801 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339558 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-podinfo\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.339801 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339565 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-istio-envoy\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.339801 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339573 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28620c-289d-4bd3-ad35-a96059384d6a-credential-socket\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.339801 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.339581 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28620c-289d-4bd3-ad35-a96059384d6a-istiod-ca-cert\") on node \"ip-10-0-138-145.ec2.internal\" DevicePath \"\"" Jul 06 13:30:24.848164 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.848129 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerID="432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79" exitCode=0 Jul 06 13:30:24.848326 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.848172 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" Jul 06 13:30:24.848326 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.848219 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" event={"ID":"3f28620c-289d-4bd3-ad35-a96059384d6a","Type":"ContainerDied","Data":"432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79"} Jul 06 13:30:24.848326 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.848273 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782" event={"ID":"3f28620c-289d-4bd3-ad35-a96059384d6a","Type":"ContainerDied","Data":"fdc22935036b2065adeeb349bb97091f3722d2737f5de53fab2c15ec423aadab"} Jul 06 13:30:24.848326 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.848304 2568 scope.go:117] "RemoveContainer" containerID="432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79" Jul 06 13:30:24.859708 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.859685 2568 scope.go:117] "RemoveContainer" containerID="79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145" Jul 06 13:30:24.867677 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.867650 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782"] Jul 06 13:30:24.868698 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.868678 2568 scope.go:117] "RemoveContainer" containerID="432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79" Jul 06 13:30:24.868999 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:30:24.868977 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79\": container with ID starting with 432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79 not found: ID does not exist" containerID="432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79" Jul 06 13:30:24.869086 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.869014 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79"} err="failed to get container status \"432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79\": rpc error: code = NotFound desc = could not find container \"432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79\": container with ID starting with 432db079e25839de14a0d743c886e0141b19ef4f8f8b79e96ecb5d8c0d604f79 not found: ID does not exist" Jul 06 13:30:24.869086 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.869039 2568 scope.go:117] "RemoveContainer" containerID="79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145" Jul 06 13:30:24.869324 ip-10-0-138-145 kubenswrapper[2568]: E0706 13:30:24.869298 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145\": container with ID starting with 79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145 not found: ID does not exist" containerID="79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145" Jul 06 13:30:24.869387 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.869335 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145"} err="failed to get container status \"79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145\": rpc error: code = NotFound desc = could not find container \"79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145\": container with ID starting with 79f3261660ffa613e073086e3e3ab7ccf18b241bcf1c711d717c078fbce4a145 not found: ID does not exist" Jul 06 13:30:24.872426 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:24.872404 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-164b27-openshift-default-5668bb654c-w9782"] Jul 06 13:30:26.595792 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:26.595752 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" path="/var/lib/kubelet/pods/3f28620c-289d-4bd3-ad35-a96059384d6a/volumes" Jul 06 13:30:32.813813 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:32.813784 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722380-x64h8_da67aed7-44a3-45f2-95d7-0a44bae09e18/cleanup/0.log" Jul 06 13:30:33.813702 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:33.813675 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58899bc45d-zc6fj_d9a8651c-19b3-44f2-9466-b217500b3b8c/copy-manifests/0.log" Jul 06 13:30:33.834547 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:33.834522 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58899bc45d-zc6fj_d9a8651c-19b3-44f2-9466-b217500b3b8c/manager/0.log" Jul 06 13:30:34.764584 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:34.764553 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2_f9100767-6c78-4bea-8152-c15731093106/util/0.log" Jul 06 13:30:34.776071 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:34.776038 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2_f9100767-6c78-4bea-8152-c15731093106/pull/0.log" Jul 06 13:30:34.787845 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:34.787820 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2_f9100767-6c78-4bea-8152-c15731093106/extract/0.log" Jul 06 13:30:34.915436 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:34.915408 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69_dcdb7470-71d1-4292-b587-a09ff2e027fb/pull/0.log" Jul 06 13:30:34.929996 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:34.929975 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69_dcdb7470-71d1-4292-b587-a09ff2e027fb/extract/0.log" Jul 06 13:30:34.939452 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:34.939438 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69_dcdb7470-71d1-4292-b587-a09ff2e027fb/util/0.log" Jul 06 13:30:35.061746 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.061669 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6_b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b/util/0.log" Jul 06 13:30:35.077535 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.077510 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6_b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b/pull/0.log" Jul 06 13:30:35.085964 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.085948 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6_b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b/extract/0.log" Jul 06 13:30:35.200011 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.199984 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m_9e665150-004d-4d22-a98a-40ffeb477e21/util/0.log" Jul 06 13:30:35.208212 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.208178 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m_9e665150-004d-4d22-a98a-40ffeb477e21/pull/0.log" Jul 06 13:30:35.216099 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.216079 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m_9e665150-004d-4d22-a98a-40ffeb477e21/extract/0.log" Jul 06 13:30:35.604385 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.604358 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gb27x_81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f/manager/0.log" Jul 06 13:30:35.720989 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.720964 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vb42s_0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd/kuadrant-console-plugin/0.log" Jul 06 13:30:35.851834 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.851802 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t442n_b00e504e-6271-40ac-8dd4-7445da038b47/registry-server/0.log" Jul 06 13:30:35.988652 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:35.988554 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xl8cr_06b3365e-a347-45fd-95ae-f16d30afc89e/manager/0.log" Jul 06 13:30:36.266797 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:36.266716 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7b9lf_8e086120-79aa-44ef-880c-936bded62191/manager/0.log" Jul 06 13:30:36.662875 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:36.662844 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-656c84dccdvvk4m_08f5adc1-7670-4b9a-9619-f8add9ce4a79/istio-proxy/0.log" Jul 06 13:30:37.162074 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:37.162043 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d7b8c9489-rn98c_e0a163c8-f10d-4f9d-8c64-e1b1fd92a185/istio-proxy/1.log" Jul 06 13:30:38.007597 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:38.007566 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-z6z55_22067628-2cb6-4c99-b404-7d4ccd6231df/main/0.log" Jul 06 13:30:38.230445 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:38.230415 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7vz2n_7e8ebc15-fde1-468d-afdd-7cd26b33d550/main/0.log" Jul 06 13:30:38.481462 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:38.481435 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-4wwj2_ec235086-9e05-44b4-b155-27ace499e6a0/main/0.log" Jul 06 13:30:45.681354 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:45.681324 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gc9tb_46062402-20c6-4730-8520-b1a94ef3c11a/global-pull-secret-syncer/0.log" Jul 06 13:30:45.934306 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:45.934214 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r7g6r_db4973a1-3cc2-4a94-aa2b-18d04d82a7d9/konnectivity-agent/0.log" Jul 06 13:30:46.059099 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:46.059068 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-145.ec2.internal_36c64bbcbb7e33685ecee28ec1d660b6/haproxy/0.log" Jul 06 13:30:50.245988 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.245957 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2_f9100767-6c78-4bea-8152-c15731093106/extract/0.log" Jul 06 13:30:50.294375 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.294346 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2_f9100767-6c78-4bea-8152-c15731093106/util/0.log" Jul 06 13:30:50.331833 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.331803 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759l8jz2_f9100767-6c78-4bea-8152-c15731093106/pull/0.log" Jul 06 13:30:50.374721 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.374696 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69_dcdb7470-71d1-4292-b587-a09ff2e027fb/extract/0.log" Jul 06 13:30:50.422599 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.422572 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69_dcdb7470-71d1-4292-b587-a09ff2e027fb/util/0.log" Jul 06 13:30:50.455543 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.455519 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d2p69_dcdb7470-71d1-4292-b587-a09ff2e027fb/pull/0.log" Jul 06 13:30:50.488465 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.488438 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6_b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b/extract/0.log" Jul 06 13:30:50.523817 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.523748 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6_b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b/util/0.log" Jul 06 13:30:50.557908 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.557881 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gplq6_b31d65be-4bf7-45dc-ba7f-0fe99a0c6d0b/pull/0.log" Jul 06 13:30:50.605495 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.605467 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m_9e665150-004d-4d22-a98a-40ffeb477e21/extract/0.log" Jul 06 13:30:50.650655 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.650604 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m_9e665150-004d-4d22-a98a-40ffeb477e21/util/0.log" Jul 06 13:30:50.688579 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:50.688553 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1zqh4m_9e665150-004d-4d22-a98a-40ffeb477e21/pull/0.log" Jul 06 13:30:51.170971 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:51.170938 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gb27x_81d84d1f-8a68-4bd2-86dc-b9f9236a0d4f/manager/0.log" Jul 06 13:30:51.205748 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:51.205722 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vb42s_0f7f0f78-9f0c-47ea-8ee5-cea7d1ad8efd/kuadrant-console-plugin/0.log" Jul 06 13:30:51.259233 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:51.259203 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-t442n_b00e504e-6271-40ac-8dd4-7445da038b47/registry-server/0.log" Jul 06 13:30:51.440273 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:51.440177 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xl8cr_06b3365e-a347-45fd-95ae-f16d30afc89e/manager/0.log" Jul 06 13:30:51.598651 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:51.598600 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7b9lf_8e086120-79aa-44ef-880c-936bded62191/manager/0.log" Jul 06 13:30:53.772480 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:53.772450 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gfz7x_721ec354-d37f-4094-8bb8-f31468932417/kube-state-metrics/0.log" Jul 06 13:30:53.828727 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:53.828696 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gfz7x_721ec354-d37f-4094-8bb8-f31468932417/kube-rbac-proxy-main/0.log" Jul 06 13:30:53.875594 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:53.875567 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gfz7x_721ec354-d37f-4094-8bb8-f31468932417/kube-rbac-proxy-self/0.log" Jul 06 13:30:53.967500 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:53.967473 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-6sqlw_3b310581-9dde-4970-9268-634eded04b9f/monitoring-plugin/0.log" Jul 06 13:30:54.165258 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:54.165208 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fvj5b_b1c83a55-eedf-424d-b7f1-051e78ae51ba/node-exporter/0.log" Jul 06 13:30:54.202225 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:54.202197 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fvj5b_b1c83a55-eedf-424d-b7f1-051e78ae51ba/kube-rbac-proxy/0.log" Jul 06 13:30:54.245339 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:54.245313 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fvj5b_b1c83a55-eedf-424d-b7f1-051e78ae51ba/init-textfile/0.log" Jul 06 13:30:54.738189 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:54.738159 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vf4zf_16d7bc32-c1ca-46d1-a368-8efd579219a4/prometheus-operator/0.log" Jul 06 13:30:54.763112 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:54.763081 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vf4zf_16d7bc32-c1ca-46d1-a368-8efd579219a4/kube-rbac-proxy/0.log" Jul 06 13:30:55.446901 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.446869 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4"] Jul 06 13:30:55.447397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.447262 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e447da-fcdd-4dc6-94f7-db0b8d97f37d" containerName="main" Jul 06 13:30:55.447397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.447278 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e447da-fcdd-4dc6-94f7-db0b8d97f37d" containerName="main" Jul 06 13:30:55.447397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.447290 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" Jul 06 13:30:55.447397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.447296 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" Jul 06 13:30:55.447397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.447381 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e447da-fcdd-4dc6-94f7-db0b8d97f37d" containerName="main" Jul 06 13:30:55.447397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.447389 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" Jul 06 13:30:55.447397 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.447396 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f28620c-289d-4bd3-ad35-a96059384d6a" containerName="istio-proxy" Jul 06 13:30:55.450795 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.450780 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.456268 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.456224 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89xhc\"/\"openshift-service-ca.crt\"" Jul 06 13:30:55.456492 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.456357 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-89xhc\"/\"default-dockercfg-bwj8j\"" Jul 06 13:30:55.458046 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.458031 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89xhc\"/\"kube-root-ca.crt\"" Jul 06 13:30:55.465171 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.465145 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4"] Jul 06 13:30:55.621898 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.621865 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-proc\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.622105 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.621936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-sys\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.622105 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.621986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-lib-modules\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.622105 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.622062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-podres\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.622105 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.622081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmf6\" (UniqueName: \"kubernetes.io/projected/59940417-47aa-4423-9a2f-0db9b9f3ead5-kube-api-access-zlmf6\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.722862 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.722752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-sys\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.722862 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.722839 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-lib-modules\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.723104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.722902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-podres\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.723104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.722918 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-sys\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.723104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.722984 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmf6\" (UniqueName: \"kubernetes.io/projected/59940417-47aa-4423-9a2f-0db9b9f3ead5-kube-api-access-zlmf6\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.723104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.723041 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-podres\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.723104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.723044 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-proc\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.723104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.723070 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-lib-modules\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.723104 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.723083 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59940417-47aa-4423-9a2f-0db9b9f3ead5-proc\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.732025 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.732002 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmf6\" (UniqueName: \"kubernetes.io/projected/59940417-47aa-4423-9a2f-0db9b9f3ead5-kube-api-access-zlmf6\") pod \"perf-node-gather-daemonset-8slk4\" (UID: \"59940417-47aa-4423-9a2f-0db9b9f3ead5\") " pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.760528 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.760504 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:55.899655 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:55.899613 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4"] Jul 06 13:30:55.901299 ip-10-0-138-145 kubenswrapper[2568]: W0706 13:30:55.901271 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod59940417_47aa_4423_9a2f_0db9b9f3ead5.slice/crio-bca3cae82d94d786b350da06168b8e03905665ca0bd86725cdfa4551eb4d8edc WatchSource:0}: Error finding container bca3cae82d94d786b350da06168b8e03905665ca0bd86725cdfa4551eb4d8edc: Status 404 returned error can't find the container with id bca3cae82d94d786b350da06168b8e03905665ca0bd86725cdfa4551eb4d8edc Jul 06 13:30:56.014908 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:56.014835 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" event={"ID":"59940417-47aa-4423-9a2f-0db9b9f3ead5","Type":"ContainerStarted","Data":"8d26e7c7a131363bd9b8e819e55f35411f53ff6a1bd104093e38496e2d4b6a92"} Jul 06 13:30:56.014908 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:56.014892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" event={"ID":"59940417-47aa-4423-9a2f-0db9b9f3ead5","Type":"ContainerStarted","Data":"bca3cae82d94d786b350da06168b8e03905665ca0bd86725cdfa4551eb4d8edc"} Jul 06 13:30:56.015052 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:56.015020 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:30:56.034046 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:56.034003 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" podStartSLOduration=1.03398975 podStartE2EDuration="1.03398975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:30:56.032514325 +0000 UTC m=+2639.973355206" watchObservedRunningTime="2026-07-06 13:30:56.03398975 +0000 UTC m=+2639.974830608" Jul 06 13:30:58.039278 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:58.039250 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-l5mj5_826fcfd6-3c84-4370-a599-5862b66e91f5/volume-data-source-validator/0.log" Jul 06 13:30:59.184473 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:59.184442 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fvt4q_3aba33d4-4f96-4774-8a16-cf7c2e4cead3/dns/0.log" Jul 06 13:30:59.211402 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:59.211377 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fvt4q_3aba33d4-4f96-4774-8a16-cf7c2e4cead3/kube-rbac-proxy/0.log" Jul 06 13:30:59.422103 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:30:59.422076 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbb7x_d2168b17-6206-410a-a930-f665ddda267a/dns-node-resolver/0.log" Jul 06 13:31:00.010566 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:00.010536 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75d59b4dc6-b65v7_12c955fc-4b31-41c1-999d-b5191af3b226/registry/0.log" Jul 06 13:31:00.099575 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:00.099536 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9x5md_efdb6ed1-7693-4154-8cd4-94005582afff/node-ca/0.log" Jul 06 13:31:01.154184 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:01.154143 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-656c84dccdvvk4m_08f5adc1-7670-4b9a-9619-f8add9ce4a79/istio-proxy/0.log" Jul 06 13:31:01.428424 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:01.428320 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d7b8c9489-rn98c_e0a163c8-f10d-4f9d-8c64-e1b1fd92a185/istio-proxy/0.log" Jul 06 13:31:01.481561 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:01.481524 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d7b8c9489-rn98c_e0a163c8-f10d-4f9d-8c64-e1b1fd92a185/istio-proxy/1.log" Jul 06 13:31:02.030072 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:02.030039 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-89xhc/perf-node-gather-daemonset-8slk4" Jul 06 13:31:02.460405 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:02.460375 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8fp5_77930e36-25bd-46b1-9808-8c87f40fc07e/serve-healthcheck-canary/0.log" Jul 06 13:31:03.046698 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:03.046662 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xnzpc_3ec35516-c4f2-40e3-bbae-fb91296ddf06/insights-operator/0.log" Jul 06 13:31:03.047674 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:03.047648 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xnzpc_3ec35516-c4f2-40e3-bbae-fb91296ddf06/insights-operator/1.log" Jul 06 13:31:03.179813 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:03.179782 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lwksb_957f84d8-8eb9-471b-9ead-476eed6c7835/kube-rbac-proxy/0.log" Jul 06 13:31:03.206381 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:03.206353 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lwksb_957f84d8-8eb9-471b-9ead-476eed6c7835/exporter/0.log" Jul 06 13:31:03.245665 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:03.245619 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lwksb_957f84d8-8eb9-471b-9ead-476eed6c7835/extractor/0.log" Jul 06 13:31:05.998594 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:05.998512 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-656c84dccdvvk4m_08f5adc1-7670-4b9a-9619-f8add9ce4a79/istio-proxy/0.log" Jul 06 13:31:06.297560 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:06.297475 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722380-x64h8_da67aed7-44a3-45f2-95d7-0a44bae09e18/cleanup/0.log" Jul 06 13:31:06.326771 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:06.326741 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d7b8c9489-rn98c_e0a163c8-f10d-4f9d-8c64-e1b1fd92a185/istio-proxy/0.log" Jul 06 13:31:06.378700 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:06.378668 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d7b8c9489-rn98c_e0a163c8-f10d-4f9d-8c64-e1b1fd92a185/istio-proxy/1.log" Jul 06 13:31:06.991376 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:06.991291 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58899bc45d-zc6fj_d9a8651c-19b3-44f2-9466-b217500b3b8c/manager/0.log" Jul 06 13:31:07.012705 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:07.012683 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58899bc45d-zc6fj_d9a8651c-19b3-44f2-9466-b217500b3b8c/copy-manifests/0.log" Jul 06 13:31:07.668333 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:07.668295 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-656c84dccdvvk4m_08f5adc1-7670-4b9a-9619-f8add9ce4a79/istio-proxy/0.log" Jul 06 13:31:07.918990 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:07.918897 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d7b8c9489-rn98c_e0a163c8-f10d-4f9d-8c64-e1b1fd92a185/istio-proxy/0.log" Jul 06 13:31:07.977933 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:07.977899 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d7b8c9489-rn98c_e0a163c8-f10d-4f9d-8c64-e1b1fd92a185/istio-proxy/1.log" Jul 06 13:31:08.446467 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:08.446428 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67468f6576-rkbwh_fa14f5a6-3e90-41f2-9e4b-48d142f1dc06/manager/0.log" Jul 06 13:31:13.505347 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:13.505309 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-s4jrx_3772227c-7e8d-435a-8e31-9043a3903802/migrator/0.log" Jul 06 13:31:13.529867 ip-10-0-138-145 kubenswrapper[2568]: I0706 13:31:13.529823 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-s4jrx_3772227c-7e8d-435a-8e31-9043a3903802/graceful-termination/0.log"