Jul 03 15:11:07.798210 ip-10-0-143-112 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 03 15:11:07.798225 ip-10-0-143-112 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 03 15:11:07.798234 ip-10-0-143-112 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 03 15:11:07.798551 ip-10-0-143-112 systemd[1]: Failed to start Kubernetes Kubelet. Jul 03 15:11:17.824016 ip-10-0-143-112 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 03 15:11:17.824032 ip-10-0-143-112 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 86856f3d1a8d440c91b7c636f81bd364 -- Jul 03 15:13:14.005308 ip-10-0-143-112 systemd[1]: Starting Kubernetes Kubelet... Jul 03 15:13:15.987514 ip-10-0-143-112 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 15:13:15.987514 ip-10-0-143-112 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 03 15:13:15.987514 ip-10-0-143-112 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 15:13:15.987514 ip-10-0-143-112 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 03 15:13:15.987514 ip-10-0-143-112 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 15:13:15.988896 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.988812 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 03 15:13:15.993203 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993186 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 15:13:15.993203 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993203 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993207 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993211 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993213 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993216 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993218 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993221 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993224 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993226 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993229 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993232 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993234 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993244 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993247 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993250 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993253 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993256 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993258 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993261 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993263 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 15:13:15.993271 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993266 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993268 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993271 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993276 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993280 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993283 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993286 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993289 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993292 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993294 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993297 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993299 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993306 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993308 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993311 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993315 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993319 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993323 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993326 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 15:13:15.993741 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993329 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993332 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993334 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993337 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993339 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993342 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993344 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993347 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993349 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993352 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993354 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993357 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993359 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993362 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993364 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993367 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993370 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993373 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993375 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993378 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 15:13:15.994220 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993380 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993383 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993386 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993389 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993393 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993396 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993399 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993402 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993404 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993407 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993409 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993413 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993415 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993418 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993420 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993423 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993425 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993427 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993430 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 15:13:15.994704 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993432 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993435 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993437 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993440 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993442 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993444 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993447 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993450 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993453 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993457 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993460 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993463 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993465 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993468 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993471 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993473 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993476 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993478 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993481 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.993484 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 15:13:15.995176 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994252 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994261 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994264 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994268 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994271 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994275 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994277 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994280 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994283 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994286 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994289 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994292 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994295 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994297 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994300 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994303 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994305 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994308 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994311 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994314 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 15:13:15.995645 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994316 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994319 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994322 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994325 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994328 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994330 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994333 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994335 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994338 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994340 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994343 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994346 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994348 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994351 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994354 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994356 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994359 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994361 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994364 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 15:13:15.996149 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994367 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994369 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994374 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994378 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994381 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994384 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994388 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994391 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994394 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994397 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994399 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994402 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994405 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994408 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994411 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994414 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994417 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994419 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994422 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 15:13:15.996612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994425 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994427 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994430 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994432 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994435 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994437 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994440 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994442 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994445 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994447 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994450 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994452 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994455 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994457 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994460 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994463 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994465 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994469 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994472 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994475 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 15:13:15.997105 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994477 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994480 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994482 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994485 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994487 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994490 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994493 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994496 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994499 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994503 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994506 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994508 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994511 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994514 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994516 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994519 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994521 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994524 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994526 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994529 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 15:13:15.997606 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994532 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994934 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994939 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994942 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994946 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994949 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994954 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994958 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994961 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994964 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994967 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994970 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994973 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994977 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994979 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.994999 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995003 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995006 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995009 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 15:13:15.998115 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995011 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995014 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995017 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995020 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995023 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995026 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995029 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995032 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995034 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995037 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995040 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995042 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995045 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995047 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995049 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995052 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995055 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995057 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995060 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995063 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 03 15:13:15.998576 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995066 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995069 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995072 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995074 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995078 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995081 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995084 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995086 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995089 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995091 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995094 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995096 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995099 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995101 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995104 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995106 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995109 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995111 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995114 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995116 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 15:13:15.999104 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995119 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995121 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995124 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995126 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995129 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995131 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995134 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995138 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995141 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995144 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995147 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995149 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995153 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995156 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995158 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995163 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995166 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995168 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995171 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995173 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 15:13:15.999613 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995176 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995178 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995181 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995185 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995188 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995191 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995193 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995196 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995198 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995201 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995204 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995206 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995209 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995212 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995214 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995217 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995219 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995221 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995224 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 15:13:16.000102 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995227 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.995229 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995300 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995309 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995318 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995324 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995347 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995354 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995364 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995370 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995374 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995377 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995381 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995384 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995387 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995390 2572 flags.go:64] FLAG: --cgroup-root="" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995393 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995396 2572 flags.go:64] FLAG: --client-ca-file="" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995399 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995402 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995410 2572 flags.go:64] FLAG: --cluster-domain="" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995413 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995416 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995419 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 03 15:13:16.000594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995423 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995427 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995429 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995432 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995436 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995439 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995442 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995445 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995448 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995451 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995455 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995458 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995461 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995464 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995468 2572 flags.go:64] FLAG: --enable-server="true" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995479 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995487 2572 flags.go:64] FLAG: --event-burst="100" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995490 2572 flags.go:64] FLAG: --event-qps="50" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995493 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995496 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995499 2572 flags.go:64] FLAG: --eviction-hard="" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995502 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995505 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995508 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995511 2572 flags.go:64] FLAG: --eviction-soft="" Jul 03 15:13:16.001192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995514 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995516 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995520 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995522 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995525 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995528 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995531 2572 flags.go:64] FLAG: --feature-gates="" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995534 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995537 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995540 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995543 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995546 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995550 2572 flags.go:64] FLAG: --help="false" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995552 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995555 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995558 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995561 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995564 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995568 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995571 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995573 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995577 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995580 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 03 15:13:16.001777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995589 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995593 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995596 2572 flags.go:64] FLAG: --kube-reserved="" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995599 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995602 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995605 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995607 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995610 2572 flags.go:64] FLAG: --lock-file="" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995613 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995616 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995619 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995628 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995631 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995634 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995637 2572 flags.go:64] FLAG: --logging-format="text" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995640 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995643 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995646 2572 flags.go:64] FLAG: --manifest-url="" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995649 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995653 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995657 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995661 2572 flags.go:64] FLAG: --max-pods="110" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995664 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995667 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995669 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 03 15:13:16.002465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995672 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995675 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995678 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995681 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995689 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995692 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995695 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995699 2572 flags.go:64] FLAG: --pod-cidr="" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995704 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995709 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995712 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995715 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995717 2572 flags.go:64] FLAG: --port="10250" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995720 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995723 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-065ee41c4563241c9" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995726 2572 flags.go:64] FLAG: --qos-reserved="" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995729 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995732 2572 flags.go:64] FLAG: --register-node="true" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995735 2572 flags.go:64] FLAG: --register-with-taints="" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995738 2572 flags.go:64] FLAG: --registry-burst="10" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995742 2572 flags.go:64] FLAG: --registry-qps="5" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995744 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995747 2572 flags.go:64] FLAG: --reserved-memory="" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995751 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995754 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 03 15:13:16.003073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995756 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995759 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995762 2572 flags.go:64] FLAG: --runonce="false" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995765 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995768 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995771 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995773 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995776 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995779 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995782 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995785 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995788 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995790 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995794 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995797 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995800 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995803 2572 flags.go:64] FLAG: --system-cgroups="" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995806 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995811 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995814 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995817 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995822 2572 flags.go:64] FLAG: --tls-min-version="" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995824 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995827 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995830 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 03 15:13:16.003669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995833 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995836 2572 flags.go:64] FLAG: --v="2" Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995840 2572 flags.go:64] FLAG: --version="false" Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995843 2572 flags.go:64] FLAG: --vmodule="" Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995847 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.995850 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996009 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996013 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996017 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996020 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996023 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996026 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996028 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996031 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996034 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996036 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996039 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996042 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996044 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996048 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996052 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 15:13:16.004319 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996055 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996058 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996061 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996069 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996072 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996075 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996077 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996080 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996082 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996085 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996087 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996090 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996093 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996095 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996098 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996101 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996103 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996106 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996108 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996111 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 15:13:16.004822 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996113 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996116 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996118 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996121 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996123 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996126 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996129 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996131 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996134 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996136 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996138 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996141 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996143 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996146 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996148 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996151 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996159 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996162 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996164 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996167 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 15:13:16.005333 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996169 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996172 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996174 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996177 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996179 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996182 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996184 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996187 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996189 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996191 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996194 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996198 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996201 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996203 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996206 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996208 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996212 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996216 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996220 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 15:13:16.005867 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996223 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996226 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996229 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996231 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996234 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996239 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996241 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996244 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996247 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996249 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996257 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996260 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996263 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996265 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996268 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996270 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996273 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996275 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996278 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 15:13:16.006618 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996280 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996283 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996285 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996287 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996290 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:15.996294 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:15.996935 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.003470 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.003486 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.003507 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.003514 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003581 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003587 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003590 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 15:13:16.007199 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003593 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003596 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003598 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003601 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003604 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003607 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003609 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003612 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003614 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003617 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003620 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003622 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003625 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003627 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003630 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003632 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003635 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003638 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003640 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 15:13:16.007758 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003643 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003646 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003648 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003651 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003653 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003656 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003659 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003663 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003666 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003669 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003671 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003674 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003678 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003682 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003685 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003688 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003691 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003694 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003697 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 15:13:16.008344 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003700 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003703 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003706 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003711 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003714 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003718 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003722 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003727 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003731 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003735 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003741 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003745 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003748 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003751 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003754 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003756 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003759 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003761 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003765 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 15:13:16.008799 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003769 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003772 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003776 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003778 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003781 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003784 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003786 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003789 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003791 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003794 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003796 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003799 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003801 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003804 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003806 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003809 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003811 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003814 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003817 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003819 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 15:13:16.009293 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003822 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003825 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003827 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003830 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003832 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003835 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003837 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003840 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003842 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003845 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003847 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003850 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003853 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003856 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003858 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003862 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003864 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003867 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 15:13:16.009767 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.003869 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.003874 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004007 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004012 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004015 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004019 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004024 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004026 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004030 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004033 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004036 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004039 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004041 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004045 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 15:13:16.010258 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004048 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004050 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004053 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004056 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004058 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004061 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004063 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004066 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004068 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004071 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004074 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004077 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004080 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004083 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004086 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004089 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004093 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004096 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004099 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 15:13:16.010604 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004101 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004104 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004107 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004109 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004112 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004114 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004117 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004119 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004121 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004124 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004126 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004129 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004131 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004134 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004137 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004139 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004142 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004144 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004147 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004149 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 15:13:16.011129 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004152 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004154 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004157 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004159 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004162 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004164 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004167 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004170 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004172 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004175 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004178 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004180 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004183 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004185 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004188 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004190 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004192 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004195 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004197 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004200 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 15:13:16.011621 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004202 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004205 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004208 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004211 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004215 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004218 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004220 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004223 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004225 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004228 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004230 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004233 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004235 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004238 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004240 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004242 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004245 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004248 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004250 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004253 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 15:13:16.012126 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004255 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004257 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004260 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004263 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004265 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004268 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004270 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:16.004272 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.004277 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.004392 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.007800 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.008977 2572 server.go:1019] "Starting client certificate rotation" Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.009084 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 15:13:16.012612 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.009116 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 15:13:16.012930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.011543 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 15:13:16.014060 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.014007 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 15:13:16.023436 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.023414 2572 log.go:25] "Validated CRI v1 runtime API" Jul 03 15:13:16.026518 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.026493 2572 log.go:25] "Validated CRI v1 image API" Jul 03 15:13:16.027328 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.027315 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 03 15:13:16.029315 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.029293 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 95b853f4-c920-457f-9369-54cd03ea7bcb:/dev/nvme0n1p3 b0aed73e-36b3-4a84-9f58-e47f5e1f2a23:/dev/nvme0n1p4] Jul 03 15:13:16.029371 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.029315 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 03 15:13:16.035199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.035073 2572 manager.go:216] Machine: {Timestamp:2026-07-03 15:13:16.033858003 +0000 UTC m=+0.266290883 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102410 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc6e5fffa487ebee42b1a4a5b7155 SystemUUID:ec2fc6e5-fffa-487e-bee4-2b1a4a5b7155 BootID:86856f3d-1a8d-440c-91b7-c636f81bd364 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:08:4b:5d:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:08:4b:5d:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:21:c4:06:bc:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 03 15:13:16.035199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.035186 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 03 15:13:16.035319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.035278 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 03 15:13:16.037665 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.037646 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 15:13:16.038049 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.038024 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 03 15:13:16.038239 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.038054 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-112.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 03 15:13:16.038705 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.038695 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 03 15:13:16.038738 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.038708 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 03 15:13:16.038738 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.038715 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 15:13:16.039333 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.039323 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 15:13:16.040465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.040453 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 03 15:13:16.044415 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.044402 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 03 15:13:16.044719 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.044708 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 03 15:13:16.044757 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.044740 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 03 15:13:16.044786 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.044763 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 03 15:13:16.044786 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.044783 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 03 15:13:16.044865 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.044798 2572 kubelet.go:398] "Adding apiserver pod source" Jul 03 15:13:16.044865 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.044808 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 03 15:13:16.045255 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.045239 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 15:13:16.045325 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.045262 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 15:13:16.048479 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.048447 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 03 15:13:16.050402 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.050387 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 03 15:13:16.050449 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.050419 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 03 15:13:16.051448 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051437 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051453 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051459 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051465 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051474 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051480 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051489 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051494 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051501 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 03 15:13:16.051504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051507 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 03 15:13:16.051729 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051516 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 03 15:13:16.051729 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.051521 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 03 15:13:16.052084 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.052073 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 03 15:13:16.052084 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.052083 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 03 15:13:16.055513 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.055493 2572 server.go:1268] "Started kubelet" Jul 03 15:13:16.055686 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.055636 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 03 15:13:16.055983 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.055714 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 03 15:13:16.056099 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.055770 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 03 15:13:16.056351 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.056319 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 03 15:13:16.056442 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.056356 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 03 15:13:16.056499 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.056455 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-112.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 03 15:13:16.056571 ip-10-0-143-112 systemd[1]: Started Kubernetes Kubelet. Jul 03 15:13:16.057858 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.057841 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 03 15:13:16.060291 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.060273 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 03 15:13:16.061035 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.061011 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 03 15:13:16.061354 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.061341 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mbzbw" Jul 03 15:13:16.064135 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.064121 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 03 15:13:16.064199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.064137 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 03 15:13:16.064265 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.064193 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.064371 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.063570 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-112.ec2.internal.18bed05c00decda6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-112.ec2.internal,UID:ip-10-0-143-112.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-112.ec2.internal,},FirstTimestamp:2026-07-03 15:13:16.055465382 +0000 UTC m=+0.287898262,LastTimestamp:2026-07-03 15:13:16.055465382 +0000 UTC m=+0.287898262,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-112.ec2.internal,}" Jul 03 15:13:16.064631 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.064617 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 03 15:13:16.064705 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.064632 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 03 15:13:16.064756 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.064619 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 03 15:13:16.064806 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.064781 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 03 15:13:16.064806 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.064789 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 03 15:13:16.065350 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.065332 2572 factory.go:55] Registering systemd factory Jul 03 15:13:16.065488 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.065425 2572 factory.go:223] Registration of the systemd container factory successfully Jul 03 15:13:16.065682 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.065667 2572 factory.go:153] Registering CRI-O factory Jul 03 15:13:16.065747 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.065684 2572 factory.go:223] Registration of the crio container factory successfully Jul 03 15:13:16.065747 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.065738 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 03 15:13:16.065852 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.065761 2572 factory.go:103] Registering Raw factory Jul 03 15:13:16.065852 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.065777 2572 manager.go:1196] Started watching for new ooms in manager Jul 03 15:13:16.067069 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.066350 2572 manager.go:349] Starting recovery of all containers Jul 03 15:13:16.067241 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.067189 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 03 15:13:16.067241 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.067191 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 03 15:13:16.068969 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.068945 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mbzbw" Jul 03 15:13:16.078940 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.078773 2572 manager.go:354] Recovery completed Jul 03 15:13:16.083372 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.083355 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 15:13:16.085314 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.085298 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory" Jul 03 15:13:16.085378 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.085333 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 15:13:16.085378 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.085344 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID" Jul 03 15:13:16.085842 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.085826 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 03 15:13:16.085842 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.085839 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 03 15:13:16.086006 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.085840 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 15:13:16.086006 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.085855 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 03 15:13:16.086553 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.086537 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory" Jul 03 15:13:16.086611 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.086565 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 15:13:16.086611 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.086576 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID" Jul 03 15:13:16.087973 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.087961 2572 policy_none.go:49] "None policy: Start" Jul 03 15:13:16.088025 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.087980 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 03 15:13:16.088025 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.088004 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 03 15:13:16.089981 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.089968 2572 policy_none.go:47] "Start" Jul 03 15:13:16.120536 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.120521 2572 manager.go:337] "Starting Device Plugin manager" Jul 03 15:13:16.120636 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.120567 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 03 15:13:16.120636 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.120580 2572 server.go:87] "Starting device plugin registration server" Jul 03 15:13:16.120856 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.120843 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 03 15:13:16.120916 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.120858 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 03 15:13:16.121024 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.121010 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 03 15:13:16.121101 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.121086 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 03 15:13:16.121101 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.121101 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 03 15:13:16.121584 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.121563 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 03 15:13:16.121641 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.121620 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.221336 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.221273 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 15:13:16.223543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.222390 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory" Jul 03 15:13:16.223543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.222422 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 15:13:16.223543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.222432 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID" Jul 03 15:13:16.223543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.222459 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.223543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.223050 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 03 15:13:16.224455 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.224436 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 03 15:13:16.224548 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.224470 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 03 15:13:16.224548 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.224491 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 03 15:13:16.224548 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.224527 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 03 15:13:16.226723 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.226705 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 15:13:16.230752 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.230737 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.230826 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.230759 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-112.ec2.internal\": node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.271602 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.271578 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.325151 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.325039 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"] Jul 03 15:13:16.325263 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.325196 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 15:13:16.326314 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.326297 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory" Jul 03 15:13:16.326383 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.326328 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 15:13:16.326383 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.326341 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID" Jul 03 15:13:16.328722 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.328709 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 15:13:16.328839 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.328827 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.328875 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.328858 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 15:13:16.329476 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.329462 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory" Jul 03 15:13:16.329476 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.329486 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 15:13:16.329589 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.329500 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID" Jul 03 15:13:16.329589 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.329534 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory" Jul 03 15:13:16.329589 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.329550 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 15:13:16.329589 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.329562 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID" Jul 03 15:13:16.331774 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.331758 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.331851 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.331789 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 15:13:16.332569 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.332555 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory" Jul 03 15:13:16.332644 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.332580 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 15:13:16.332644 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.332590 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID" Jul 03 15:13:16.360664 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.360643 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-112.ec2.internal\" not found" node="ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.364264 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.364247 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-112.ec2.internal\" not found" node="ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.366458 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.366441 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e9af105ef1fd68636e93b862a9203b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"c7e9af105ef1fd68636e93b862a9203b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.366524 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.366464 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e9af105ef1fd68636e93b862a9203b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"c7e9af105ef1fd68636e93b862a9203b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.366524 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.366482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79a3a725957a215de175bf549eccdfcf-config\") pod \"kube-apiserver-proxy-ip-10-0-143-112.ec2.internal\" (UID: \"79a3a725957a215de175bf549eccdfcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.372560 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.372544 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.466709 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.466674 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e9af105ef1fd68636e93b862a9203b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"c7e9af105ef1fd68636e93b862a9203b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.466709 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.466706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e9af105ef1fd68636e93b862a9203b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"c7e9af105ef1fd68636e93b862a9203b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.466898 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.466726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79a3a725957a215de175bf549eccdfcf-config\") pod \"kube-apiserver-proxy-ip-10-0-143-112.ec2.internal\" (UID: \"79a3a725957a215de175bf549eccdfcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.466898 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.466768 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e9af105ef1fd68636e93b862a9203b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"c7e9af105ef1fd68636e93b862a9203b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.466898 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.466774 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e9af105ef1fd68636e93b862a9203b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"c7e9af105ef1fd68636e93b862a9203b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.466898 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.466807 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79a3a725957a215de175bf549eccdfcf-config\") pod \"kube-apiserver-proxy-ip-10-0-143-112.ec2.internal\" (UID: \"79a3a725957a215de175bf549eccdfcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.472824 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.472767 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.573586 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.573558 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.663793 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.663756 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.666980 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.666959 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" Jul 03 15:13:16.674026 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.674002 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.774557 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.774463 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.875069 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.875038 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.975710 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:16.975679 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found" Jul 03 15:13:16.998189 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:16.998165 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 15:13:17.008129 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.008103 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 03 15:13:17.008273 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.008254 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 03 15:13:17.008322 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.008284 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 03 15:13:17.045512 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.045434 2572 apiserver.go:52] "Watching apiserver" Jul 03 15:13:17.064713 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.064684 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" Jul 03 15:13:17.064841 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.064717 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 03 15:13:17.066442 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.066424 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 03 15:13:17.066798 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.066781 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-5pbkd","openshift-ovn-kubernetes/ovnkube-node-bzpv8","kube-system/konnectivity-agent-v62c6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c","openshift-cluster-node-tuning-operator/tuned-5sp5v","openshift-image-registry/node-ca-lct2j","openshift-multus/multus-additional-cni-plugins-jk58z","openshift-multus/multus-gxxq5","openshift-network-operator/iptables-alerter-j4crr","kube-system/global-pull-secret-syncer-c29qj","openshift-dns/node-resolver-lq5dz","openshift-multus/network-metrics-daemon-rjszq"] Jul 03 15:13:17.069281 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.069266 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:17.069359 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.069343 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:17.071484 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.071470 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.073570 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.073539 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-02 15:08:16 +0000 UTC" deadline="2027-12-24 21:50:22.956658158 +0000 UTC" Jul 03 15:13:17.073570 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.073565 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12942h37m5.883096106s" Jul 03 15:13:17.073975 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.073953 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.074779 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.074288 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 03 15:13:17.074779 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.074382 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 03 15:13:17.074779 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.074595 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ghdbc\"" Jul 03 15:13:17.075339 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.075321 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 03 15:13:17.075504 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.075490 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 03 15:13:17.075668 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.075656 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 03 15:13:17.075796 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.075778 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 15:13:17.075858 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.075811 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 03 15:13:17.075918 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.075903 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" Jul 03 15:13:17.077391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.077370 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dp8lw\"" Jul 03 15:13:17.077391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.077391 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.077571 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.077556 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 03 15:13:17.078480 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.078456 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 03 15:13:17.079610 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.079595 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.079799 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.079781 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gcjpp\"" Jul 03 15:13:17.080040 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.080024 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 03 15:13:17.080094 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.080061 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 03 15:13:17.080348 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.080334 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 03 15:13:17.081069 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.081053 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 15:13:17.081721 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.081710 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.082216 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.082197 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 03 15:13:17.082296 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.082220 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2wsmd\"" Jul 03 15:13:17.082296 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.082226 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 03 15:13:17.083864 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.083851 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 03 15:13:17.084433 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.084405 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.084519 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.084437 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 03 15:13:17.084573 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.084538 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-64pm7\"" Jul 03 15:13:17.084647 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.084542 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 03 15:13:17.085155 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.085142 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 15:13:17.086661 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.086646 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.087207 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.087190 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 03 15:13:17.087263 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.087209 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 03 15:13:17.087263 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.087227 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 03 15:13:17.087263 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.087235 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 03 15:13:17.087481 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.087464 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 03 15:13:17.087542 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.087499 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b2jfc\"" Jul 03 15:13:17.089061 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.089043 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.089161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.089099 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vnznm\"" Jul 03 15:13:17.089409 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.089395 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 03 15:13:17.091538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.091491 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 03 15:13:17.091642 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.091555 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.091709 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.091639 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:17.091776 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.091761 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 03 15:13:17.093528 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.093511 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 03 15:13:17.093607 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.093513 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6zjcr\"" Jul 03 15:13:17.093834 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.093816 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.095122 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.095105 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-988w6" Jul 03 15:13:17.096125 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.096111 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"] Jul 03 15:13:17.096199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.096135 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"] Jul 03 15:13:17.096255 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.096229 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:17.096325 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.096305 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:17.096547 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.096533 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5fw27\"" Jul 03 15:13:17.096598 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.096540 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 03 15:13:17.096598 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.096586 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 03 15:13:17.097968 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.097935 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 15:13:17.102036 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.102021 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-988w6" Jul 03 15:13:17.165631 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.165607 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 03 15:13:17.170145 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-etc-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170244 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170244 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-cni-netd\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170244 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-ovnkube-config\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170367 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170242 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-os-release\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.170367 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.170367 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170289 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27h5\" (UniqueName: \"kubernetes.io/projected/82d6aa1f-2b29-4328-83e5-2e961402a35f-kube-api-access-z27h5\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.170367 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170325 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysctl-conf\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.170367 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-node-log\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170568 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-run-netns\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170568 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82d6aa1f-2b29-4328-83e5-2e961402a35f-hosts-file\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.170568 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170443 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-run\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.170568 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-var-lib-kubelet\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.170568 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-host\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-os-release\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-cni-bin\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-modprobe-d\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170664 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysconfig\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170689 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-systemd-units\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170705 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03c15373-89c7-4c47-8c01-0c6498dedce5-konnectivity-ca\") pod \"konnectivity-agent-v62c6\" (UID: \"03c15373-89c7-4c47-8c01-0c6498dedce5\") " pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170721 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-kubelet-config\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170739 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj98q\" (UniqueName: \"kubernetes.io/projected/6eb9e801-5990-47ae-964f-6f8e509cffbf-kube-api-access-rj98q\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:17.170783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170769 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-registration-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170807 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-netns\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-conf-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170862 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrdj\" (UniqueName: \"kubernetes.io/projected/361fddc6-1cd2-433b-8ced-6e459e1f3094-kube-api-access-dkrdj\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-kubelet\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-systemd\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.170959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-env-overrides\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171030 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7khs\" (UniqueName: \"kubernetes.io/projected/220c038d-e52f-47b6-ab0a-3782911b6f83-kube-api-access-x7khs\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171061 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-ovn\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b762f10c-f840-4d77-9597-e77cc779b7f1-ovn-node-metrics-cert\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvq8c\" (UniqueName: \"kubernetes.io/projected/b762f10c-f840-4d77-9597-e77cc779b7f1-kube-api-access-wvq8c\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171141 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-lib-modules\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171154 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2dc4ba3-f734-4053-acee-fed33f36b791-serviceca\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171177 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/220c038d-e52f-47b6-ab0a-3782911b6f83-cni-binary-copy\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d6aa1f-2b29-4328-83e5-2e961402a35f-tmp-dir\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-tuned\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171247 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfnc\" (UniqueName: \"kubernetes.io/projected/c80039b1-5090-4dc1-a555-057c8ddb4337-kube-api-access-wjfnc\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-system-cni-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171297 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-var-lib-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c80039b1-5090-4dc1-a555-057c8ddb4337-tmp\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171365 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2dc4ba3-f734-4053-acee-fed33f36b791-host\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171378 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/361fddc6-1cd2-433b-8ced-6e459e1f3094-host-slash\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171393 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-dbus\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-sys\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171442 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-cni-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171456 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-k8s-cni-cncf-io\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.171511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-slash\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171492 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-log-socket\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171513 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-cnibin\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171527 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171541 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-cni-bin\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171577 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03c15373-89c7-4c47-8c01-0c6498dedce5-agent-certs\") pod \"konnectivity-agent-v62c6\" (UID: \"03c15373-89c7-4c47-8c01-0c6498dedce5\") " pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-cni-multus\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171650 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-ovnkube-script-lib\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171685 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-kubernetes\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171710 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-socket-dir-parent\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-multus-certs\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/361fddc6-1cd2-433b-8ced-6e459e1f3094-iptables-alerter-script\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171771 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:17.172073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgr2g\" (UniqueName: \"kubernetes.io/projected/5e33c5cd-6688-4a91-9744-a36058fdb15f-kube-api-access-hgr2g\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171800 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-device-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-sys-fs\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171891 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2cvp\" (UniqueName: \"kubernetes.io/projected/fd94fa95-3ca6-469b-93fd-3aae5929e74d-kube-api-access-c2cvp\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171920 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysctl-d\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-daemon-config\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.171998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-system-cni-dir\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-socket-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-systemd\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5s8\" (UniqueName: \"kubernetes.io/projected/b2dc4ba3-f734-4053-acee-fed33f36b791-kube-api-access-vq5s8\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172101 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-cnibin\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-kubelet\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-hostroot\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.172690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.172216 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-etc-kubernetes\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.272693 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-sys\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.272693 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-cni-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-k8s-cni-cncf-io\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-slash\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-log-socket\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272774 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-sys\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-cnibin\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272799 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-cni-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272802 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-k8s-cni-cncf-io\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272802 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-slash\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272835 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-log-socket\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272840 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-cnibin\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272880 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-cni-bin\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272908 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-cni-bin\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272925 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.272930 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272952 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03c15373-89c7-4c47-8c01-0c6498dedce5-agent-certs\") pod \"konnectivity-agent-v62c6\" (UID: \"03c15373-89c7-4c47-8c01-0c6498dedce5\") " pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.272969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-cni-multus\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273021 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-ovnkube-script-lib\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-kubernetes\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273070 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-cni-multus\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273073 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273101 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-kubernetes\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-socket-dir-parent\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-multus-certs\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/361fddc6-1cd2-433b-8ced-6e459e1f3094-iptables-alerter-script\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273245 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273261 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-multus-certs\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgr2g\" (UniqueName: \"kubernetes.io/projected/5e33c5cd-6688-4a91-9744-a36058fdb15f-kube-api-access-hgr2g\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.273529 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-device-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-sys-fs\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273359 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273369 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2cvp\" (UniqueName: \"kubernetes.io/projected/fd94fa95-3ca6-469b-93fd-3aae5929e74d-kube-api-access-c2cvp\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysctl-d\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-daemon-config\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-system-cni-dir\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-socket-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273473 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-device-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-systemd\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273272 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-socket-dir-parent\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5s8\" (UniqueName: \"kubernetes.io/projected/b2dc4ba3-f734-4053-acee-fed33f36b791-kube-api-access-vq5s8\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-cnibin\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.273576 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273624 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-kubelet\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273645 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysctl-d\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273664 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-ovnkube-script-lib\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.274352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273665 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273683 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-systemd\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.273698 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret podName:8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec nodeName:}" failed. No retries permitted until 2026-07-03 15:13:17.7736308 +0000 UTC m=+2.006063669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret") pod "global-pull-secret-syncer-c29qj" (UID: "8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec") : object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-var-lib-kubelet\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273736 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-socket-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273783 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-hostroot\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273828 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-system-cni-dir\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273740 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-hostroot\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273856 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-cnibin\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273872 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273891 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-sys-fs\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-etc-kubernetes\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273904 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/361fddc6-1cd2-433b-8ced-6e459e1f3094-iptables-alerter-script\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-etc-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273954 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-etc-kubernetes\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.273981 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-etc-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.275206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-cni-netd\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274027 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-ovnkube-config\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274033 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-os-release\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274076 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-cni-netd\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274076 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27h5\" (UniqueName: \"kubernetes.io/projected/82d6aa1f-2b29-4328-83e5-2e961402a35f-kube-api-access-z27h5\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274138 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e33c5cd-6688-4a91-9744-a36058fdb15f-os-release\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysctl-conf\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-node-log\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274194 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-run-netns\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274218 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82d6aa1f-2b29-4328-83e5-2e961402a35f-hosts-file\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274240 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-run\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-var-lib-kubelet\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-host\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-os-release\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274318 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-daemon-config\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.276035 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-cni-bin\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274371 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-modprobe-d\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274394 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-cni-bin\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysconfig\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274410 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274419 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-node-log\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274431 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-systemd-units\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274442 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysconfig\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03c15373-89c7-4c47-8c01-0c6498dedce5-konnectivity-ca\") pod \"konnectivity-agent-v62c6\" (UID: \"03c15373-89c7-4c47-8c01-0c6498dedce5\") " pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274482 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-os-release\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274502 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-run-netns\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-kubelet-config\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274548 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-host\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274557 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-ovnkube-config\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274562 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-sysctl-conf\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj98q\" (UniqueName: \"kubernetes.io/projected/6eb9e801-5990-47ae-964f-6f8e509cffbf-kube-api-access-rj98q\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274560 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82d6aa1f-2b29-4328-83e5-2e961402a35f-hosts-file\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.276843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274600 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-kubelet-config\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274604 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e33c5cd-6688-4a91-9744-a36058fdb15f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-registration-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274636 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-systemd-units\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-netns\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274659 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-run\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274663 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-modprobe-d\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274668 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-host-run-netns\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274671 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-conf-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274707 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-registration-dir\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274712 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-var-lib-kubelet\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274713 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-multus-conf-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrdj\" (UniqueName: \"kubernetes.io/projected/361fddc6-1cd2-433b-8ced-6e459e1f3094-kube-api-access-dkrdj\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274771 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-kubelet\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274785 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd94fa95-3ca6-469b-93fd-3aae5929e74d-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274793 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-systemd\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.277725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274817 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-host-kubelet\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-env-overrides\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274875 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-systemd\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7khs\" (UniqueName: \"kubernetes.io/projected/220c038d-e52f-47b6-ab0a-3782911b6f83-kube-api-access-x7khs\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-ovn\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b762f10c-f840-4d77-9597-e77cc779b7f1-ovn-node-metrics-cert\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.274978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvq8c\" (UniqueName: \"kubernetes.io/projected/b762f10c-f840-4d77-9597-e77cc779b7f1-kube-api-access-wvq8c\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275010 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-ovn\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-lib-modules\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275039 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03c15373-89c7-4c47-8c01-0c6498dedce5-konnectivity-ca\") pod \"konnectivity-agent-v62c6\" (UID: \"03c15373-89c7-4c47-8c01-0c6498dedce5\") " pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2dc4ba3-f734-4053-acee-fed33f36b791-serviceca\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/220c038d-e52f-47b6-ab0a-3782911b6f83-cni-binary-copy\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d6aa1f-2b29-4328-83e5-2e961402a35f-tmp-dir\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-tuned\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfnc\" (UniqueName: \"kubernetes.io/projected/c80039b1-5090-4dc1-a555-057c8ddb4337-kube-api-access-wjfnc\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-system-cni-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275231 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b762f10c-f840-4d77-9597-e77cc779b7f1-env-overrides\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-var-lib-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275274 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c80039b1-5090-4dc1-a555-057c8ddb4337-tmp\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275332 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2dc4ba3-f734-4053-acee-fed33f36b791-host\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/361fddc6-1cd2-433b-8ced-6e459e1f3094-host-slash\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275379 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-var-lib-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275380 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-dbus\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275443 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b762f10c-f840-4d77-9597-e77cc779b7f1-run-openvswitch\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275643 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2dc4ba3-f734-4053-acee-fed33f36b791-serviceca\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275122 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c80039b1-5090-4dc1-a555-057c8ddb4337-lib-modules\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275646 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/220c038d-e52f-47b6-ab0a-3782911b6f83-cni-binary-copy\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275679 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d6aa1f-2b29-4328-83e5-2e961402a35f-tmp-dir\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275706 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/361fddc6-1cd2-433b-8ced-6e459e1f3094-host-slash\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275750 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/220c038d-e52f-47b6-ab0a-3782911b6f83-system-cni-dir\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275773 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2dc4ba3-f734-4053-acee-fed33f36b791-host\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275792 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-dbus\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.275818 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.275870 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs podName:6eb9e801-5990-47ae-964f-6f8e509cffbf nodeName:}" failed. No retries permitted until 2026-07-03 15:13:17.775854826 +0000 UTC m=+2.008287696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs") pod "network-metrics-daemon-rjszq" (UID: "6eb9e801-5990-47ae-964f-6f8e509cffbf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.275694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:17.278818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.277295 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b762f10c-f840-4d77-9597-e77cc779b7f1-ovn-node-metrics-cert\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.279460 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.277299 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03c15373-89c7-4c47-8c01-0c6498dedce5-agent-certs\") pod \"konnectivity-agent-v62c6\" (UID: \"03c15373-89c7-4c47-8c01-0c6498dedce5\") " pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.279460 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.277523 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c80039b1-5090-4dc1-a555-057c8ddb4337-etc-tuned\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.279460 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.278033 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c80039b1-5090-4dc1-a555-057c8ddb4337-tmp\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.282465 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.282445 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 15:13:17.282465 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.282463 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 15:13:17.282605 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.282473 2572 projected.go:196] Error preparing data for projected volume kube-api-access-2bq8p for pod openshift-network-diagnostics/network-check-target-5pbkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:17.282605 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.282527 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p podName:abe62e1f-4043-470b-827e-6382ee35b8b7 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:17.782510906 +0000 UTC m=+2.014943772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2bq8p" (UniqueName: "kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p") pod "network-check-target-5pbkd" (UID: "abe62e1f-4043-470b-827e-6382ee35b8b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:17.285052 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.285026 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5s8\" (UniqueName: \"kubernetes.io/projected/b2dc4ba3-f734-4053-acee-fed33f36b791-kube-api-access-vq5s8\") pod \"node-ca-lct2j\" (UID: \"b2dc4ba3-f734-4053-acee-fed33f36b791\") " pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.286395 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.286374 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgr2g\" (UniqueName: \"kubernetes.io/projected/5e33c5cd-6688-4a91-9744-a36058fdb15f-kube-api-access-hgr2g\") pod \"multus-additional-cni-plugins-jk58z\" (UID: \"5e33c5cd-6688-4a91-9744-a36058fdb15f\") " pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.286525 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.286395 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2cvp\" (UniqueName: \"kubernetes.io/projected/fd94fa95-3ca6-469b-93fd-3aae5929e74d-kube-api-access-c2cvp\") pod \"aws-ebs-csi-driver-node-89c6c\" (UID: \"fd94fa95-3ca6-469b-93fd-3aae5929e74d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.287147 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.287123 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrdj\" (UniqueName: \"kubernetes.io/projected/361fddc6-1cd2-433b-8ced-6e459e1f3094-kube-api-access-dkrdj\") pod \"iptables-alerter-j4crr\" (UID: \"361fddc6-1cd2-433b-8ced-6e459e1f3094\") " pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.287452 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.287432 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvq8c\" (UniqueName: \"kubernetes.io/projected/b762f10c-f840-4d77-9597-e77cc779b7f1-kube-api-access-wvq8c\") pod \"ovnkube-node-bzpv8\" (UID: \"b762f10c-f840-4d77-9597-e77cc779b7f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.287982 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.287961 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27h5\" (UniqueName: \"kubernetes.io/projected/82d6aa1f-2b29-4328-83e5-2e961402a35f-kube-api-access-z27h5\") pod \"node-resolver-lq5dz\" (UID: \"82d6aa1f-2b29-4328-83e5-2e961402a35f\") " pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.288127 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.288112 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7khs\" (UniqueName: \"kubernetes.io/projected/220c038d-e52f-47b6-ab0a-3782911b6f83-kube-api-access-x7khs\") pod \"multus-gxxq5\" (UID: \"220c038d-e52f-47b6-ab0a-3782911b6f83\") " pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.288298 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.288280 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfnc\" (UniqueName: \"kubernetes.io/projected/c80039b1-5090-4dc1-a555-057c8ddb4337-kube-api-access-wjfnc\") pod \"tuned-5sp5v\" (UID: \"c80039b1-5090-4dc1-a555-057c8ddb4337\") " pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.288874 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.288859 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj98q\" (UniqueName: \"kubernetes.io/projected/6eb9e801-5990-47ae-964f-6f8e509cffbf-kube-api-access-rj98q\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:17.311156 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.311112 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e9af105ef1fd68636e93b862a9203b.slice/crio-313716e2c70c5e78dde20d46da7b8e4d73cac3463e34bb505777bd70eaff9f97 WatchSource:0}: Error finding container 313716e2c70c5e78dde20d46da7b8e4d73cac3463e34bb505777bd70eaff9f97: Status 404 returned error can't find the container with id 313716e2c70c5e78dde20d46da7b8e4d73cac3463e34bb505777bd70eaff9f97 Jul 03 15:13:17.311624 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.311604 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a3a725957a215de175bf549eccdfcf.slice/crio-dc3a3485126557667c4306d5cd823adae264b26c1d28c29196e8b710b7b39ad5 WatchSource:0}: Error finding container dc3a3485126557667c4306d5cd823adae264b26c1d28c29196e8b710b7b39ad5: Status 404 returned error can't find the container with id dc3a3485126557667c4306d5cd823adae264b26c1d28c29196e8b710b7b39ad5 Jul 03 15:13:17.317488 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.317472 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 15:13:17.396735 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.396707 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:17.405964 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.405925 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb762f10c_f840_4d77_9597_e77cc779b7f1.slice/crio-81032c13bff1cfffcd94e7c72dee0ec94bc31ec16e5a95b0dcdaf634d6e10a9f WatchSource:0}: Error finding container 81032c13bff1cfffcd94e7c72dee0ec94bc31ec16e5a95b0dcdaf634d6e10a9f: Status 404 returned error can't find the container with id 81032c13bff1cfffcd94e7c72dee0ec94bc31ec16e5a95b0dcdaf634d6e10a9f Jul 03 15:13:17.413874 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.413854 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:17.423489 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.423466 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c15373_89c7_4c47_8c01_0c6498dedce5.slice/crio-efc338af84bb4024cecd7f85bd657a687ea7e1e6a4fd168e3cd4b2ea3d3701e1 WatchSource:0}: Error finding container efc338af84bb4024cecd7f85bd657a687ea7e1e6a4fd168e3cd4b2ea3d3701e1: Status 404 returned error can't find the container with id efc338af84bb4024cecd7f85bd657a687ea7e1e6a4fd168e3cd4b2ea3d3701e1 Jul 03 15:13:17.431923 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.431905 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" Jul 03 15:13:17.439548 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.439521 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd94fa95_3ca6_469b_93fd_3aae5929e74d.slice/crio-208723ea9fcea5c89cd053dc995564b6df345b2292acefe693c57210d367b17f WatchSource:0}: Error finding container 208723ea9fcea5c89cd053dc995564b6df345b2292acefe693c57210d367b17f: Status 404 returned error can't find the container with id 208723ea9fcea5c89cd053dc995564b6df345b2292acefe693c57210d367b17f Jul 03 15:13:17.449002 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.448968 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" Jul 03 15:13:17.452795 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.452781 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lct2j" Jul 03 15:13:17.456971 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.456945 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80039b1_5090_4dc1_a555_057c8ddb4337.slice/crio-fe3b4c8ab5f16d4bf89883d6e5cbf999119795b9f7ae2cb2d90a3f2db55e131d WatchSource:0}: Error finding container fe3b4c8ab5f16d4bf89883d6e5cbf999119795b9f7ae2cb2d90a3f2db55e131d: Status 404 returned error can't find the container with id fe3b4c8ab5f16d4bf89883d6e5cbf999119795b9f7ae2cb2d90a3f2db55e131d Jul 03 15:13:17.461505 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.461488 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2dc4ba3_f734_4053_acee_fed33f36b791.slice/crio-84db60533804ca9abf85a03d724750dac08fe8716dc28df0797a793e1ef3792c WatchSource:0}: Error finding container 84db60533804ca9abf85a03d724750dac08fe8716dc28df0797a793e1ef3792c: Status 404 returned error can't find the container with id 84db60533804ca9abf85a03d724750dac08fe8716dc28df0797a793e1ef3792c Jul 03 15:13:17.474712 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.474694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jk58z" Jul 03 15:13:17.483430 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.483410 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e33c5cd_6688_4a91_9744_a36058fdb15f.slice/crio-8ce0a3e6282b02c0d795c710d2fff5ed0466a49215eefbcbe9863d1bd0a334a3 WatchSource:0}: Error finding container 8ce0a3e6282b02c0d795c710d2fff5ed0466a49215eefbcbe9863d1bd0a334a3: Status 404 returned error can't find the container with id 8ce0a3e6282b02c0d795c710d2fff5ed0466a49215eefbcbe9863d1bd0a334a3 Jul 03 15:13:17.491796 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.491779 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxxq5" Jul 03 15:13:17.499869 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.499848 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220c038d_e52f_47b6_ab0a_3782911b6f83.slice/crio-37466952ff599804361c56f57dca7dba1a530d923a66b18d5b9e25b188cd9bd2 WatchSource:0}: Error finding container 37466952ff599804361c56f57dca7dba1a530d923a66b18d5b9e25b188cd9bd2: Status 404 returned error can't find the container with id 37466952ff599804361c56f57dca7dba1a530d923a66b18d5b9e25b188cd9bd2 Jul 03 15:13:17.512460 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.512435 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j4crr" Jul 03 15:13:17.517187 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.517167 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq5dz" Jul 03 15:13:17.522148 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.522126 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361fddc6_1cd2_433b_8ced_6e459e1f3094.slice/crio-9204bbc51808d00cf5fa0091aee32aa09997e12ac964d05ef9ad988caef63e4a WatchSource:0}: Error finding container 9204bbc51808d00cf5fa0091aee32aa09997e12ac964d05ef9ad988caef63e4a: Status 404 returned error can't find the container with id 9204bbc51808d00cf5fa0091aee32aa09997e12ac964d05ef9ad988caef63e4a Jul 03 15:13:17.526475 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:17.526452 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d6aa1f_2b29_4328_83e5_2e961402a35f.slice/crio-c3bbcbdd7a793a02156ad0e8f40af39f3dd96ee60bc0224316eb2a598f6a40cc WatchSource:0}: Error finding container c3bbcbdd7a793a02156ad0e8f40af39f3dd96ee60bc0224316eb2a598f6a40cc: Status 404 returned error can't find the container with id c3bbcbdd7a793a02156ad0e8f40af39f3dd96ee60bc0224316eb2a598f6a40cc Jul 03 15:13:17.543040 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.543013 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 15:13:17.779945 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.779914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:17.780110 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.779968 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:17.780110 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.780076 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:17.780110 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.780090 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:17.780208 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.780155 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs podName:6eb9e801-5990-47ae-964f-6f8e509cffbf nodeName:}" failed. No retries permitted until 2026-07-03 15:13:18.780140085 +0000 UTC m=+3.012572955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs") pod "network-metrics-daemon-rjszq" (UID: "6eb9e801-5990-47ae-964f-6f8e509cffbf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:17.780208 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.780171 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret podName:8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec nodeName:}" failed. No retries permitted until 2026-07-03 15:13:18.780164437 +0000 UTC m=+3.012597304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret") pod "global-pull-secret-syncer-c29qj" (UID: "8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec") : object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:17.881498 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.880813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:17.881498 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.881052 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 15:13:17.881498 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.881076 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 15:13:17.881498 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.881089 2572 projected.go:196] Error preparing data for projected volume kube-api-access-2bq8p for pod openshift-network-diagnostics/network-check-target-5pbkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:17.881498 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:17.881149 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p podName:abe62e1f-4043-470b-827e-6382ee35b8b7 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:18.88113158 +0000 UTC m=+3.113564461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bq8p" (UniqueName: "kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p") pod "network-check-target-5pbkd" (UID: "abe62e1f-4043-470b-827e-6382ee35b8b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:17.963842 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:17.963810 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 15:13:18.104975 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.104917 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 15:08:17 +0000 UTC" deadline="2028-02-19 03:09:30.648445522 +0000 UTC" Jul 03 15:13:18.104975 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.104971 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14291h56m12.543477893s" Jul 03 15:13:18.227897 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.227079 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:18.227897 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.227197 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:18.227897 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.227590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:18.227897 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.227693 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:18.264336 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.264273 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq5dz" event={"ID":"82d6aa1f-2b29-4328-83e5-2e961402a35f","Type":"ContainerStarted","Data":"c3bbcbdd7a793a02156ad0e8f40af39f3dd96ee60bc0224316eb2a598f6a40cc"} Jul 03 15:13:18.271530 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.271494 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerStarted","Data":"8ce0a3e6282b02c0d795c710d2fff5ed0466a49215eefbcbe9863d1bd0a334a3"} Jul 03 15:13:18.281017 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.280968 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" event={"ID":"c80039b1-5090-4dc1-a555-057c8ddb4337","Type":"ContainerStarted","Data":"fe3b4c8ab5f16d4bf89883d6e5cbf999119795b9f7ae2cb2d90a3f2db55e131d"} Jul 03 15:13:18.302436 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.302262 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"81032c13bff1cfffcd94e7c72dee0ec94bc31ec16e5a95b0dcdaf634d6e10a9f"} Jul 03 15:13:18.314665 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.314633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" event={"ID":"c7e9af105ef1fd68636e93b862a9203b","Type":"ContainerStarted","Data":"313716e2c70c5e78dde20d46da7b8e4d73cac3463e34bb505777bd70eaff9f97"} Jul 03 15:13:18.321721 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.321689 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" event={"ID":"79a3a725957a215de175bf549eccdfcf","Type":"ContainerStarted","Data":"dc3a3485126557667c4306d5cd823adae264b26c1d28c29196e8b710b7b39ad5"} Jul 03 15:13:18.347390 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.347342 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j4crr" event={"ID":"361fddc6-1cd2-433b-8ced-6e459e1f3094","Type":"ContainerStarted","Data":"9204bbc51808d00cf5fa0091aee32aa09997e12ac964d05ef9ad988caef63e4a"} Jul 03 15:13:18.362305 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.362229 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxxq5" event={"ID":"220c038d-e52f-47b6-ab0a-3782911b6f83","Type":"ContainerStarted","Data":"37466952ff599804361c56f57dca7dba1a530d923a66b18d5b9e25b188cd9bd2"} Jul 03 15:13:18.368451 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.368299 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lct2j" event={"ID":"b2dc4ba3-f734-4053-acee-fed33f36b791","Type":"ContainerStarted","Data":"84db60533804ca9abf85a03d724750dac08fe8716dc28df0797a793e1ef3792c"} Jul 03 15:13:18.384784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.384749 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" event={"ID":"fd94fa95-3ca6-469b-93fd-3aae5929e74d","Type":"ContainerStarted","Data":"208723ea9fcea5c89cd053dc995564b6df345b2292acefe693c57210d367b17f"} Jul 03 15:13:18.409471 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.409431 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v62c6" event={"ID":"03c15373-89c7-4c47-8c01-0c6498dedce5","Type":"ContainerStarted","Data":"efc338af84bb4024cecd7f85bd657a687ea7e1e6a4fd168e3cd4b2ea3d3701e1"} Jul 03 15:13:18.790552 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.790466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:18.790719 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.790554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:18.790719 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.790712 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:18.790832 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.790773 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs podName:6eb9e801-5990-47ae-964f-6f8e509cffbf nodeName:}" failed. No retries permitted until 2026-07-03 15:13:20.790756028 +0000 UTC m=+5.023188902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs") pod "network-metrics-daemon-rjszq" (UID: "6eb9e801-5990-47ae-964f-6f8e509cffbf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:18.791814 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.791164 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:18.791814 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.791216 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret podName:8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec nodeName:}" failed. No retries permitted until 2026-07-03 15:13:20.791200552 +0000 UTC m=+5.023633423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret") pod "global-pull-secret-syncer-c29qj" (UID: "8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec") : object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:18.891870 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:18.891833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:18.892060 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.892042 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 15:13:18.892141 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.892067 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 15:13:18.892141 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.892080 2572 projected.go:196] Error preparing data for projected volume kube-api-access-2bq8p for pod openshift-network-diagnostics/network-check-target-5pbkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:18.892241 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:18.892144 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p podName:abe62e1f-4043-470b-827e-6382ee35b8b7 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:20.892126278 +0000 UTC m=+5.124559148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bq8p" (UniqueName: "kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p") pod "network-check-target-5pbkd" (UID: "abe62e1f-4043-470b-827e-6382ee35b8b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:19.105803 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:19.105593 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 15:08:17 +0000 UTC" deadline="2028-04-08 10:29:40.231879416 +0000 UTC" Jul 03 15:13:19.105803 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:19.105631 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15475h16m21.126253539s" Jul 03 15:13:19.156716 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:19.156687 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 15:13:19.224741 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:19.224706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:19.224919 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:19.224863 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:20.228096 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:20.227829 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:20.228658 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.228204 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:20.228658 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:20.228271 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:20.228658 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.228336 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:20.809509 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:20.809471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:20.809688 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:20.809547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:20.809688 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.809678 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:20.809806 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.809724 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret podName:8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec nodeName:}" failed. No retries permitted until 2026-07-03 15:13:24.809711738 +0000 UTC m=+9.042144609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret") pod "global-pull-secret-syncer-c29qj" (UID: "8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec") : object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:20.810111 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.810095 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:20.810185 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.810146 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs podName:6eb9e801-5990-47ae-964f-6f8e509cffbf nodeName:}" failed. No retries permitted until 2026-07-03 15:13:24.810133819 +0000 UTC m=+9.042566703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs") pod "network-metrics-daemon-rjszq" (UID: "6eb9e801-5990-47ae-964f-6f8e509cffbf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:20.910757 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:20.910719 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:20.910959 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.910912 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 15:13:20.910959 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.910930 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 15:13:20.910959 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.910939 2572 projected.go:196] Error preparing data for projected volume kube-api-access-2bq8p for pod openshift-network-diagnostics/network-check-target-5pbkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:20.911143 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:20.910981 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p podName:abe62e1f-4043-470b-827e-6382ee35b8b7 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:24.910968956 +0000 UTC m=+9.143401824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bq8p" (UniqueName: "kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p") pod "network-check-target-5pbkd" (UID: "abe62e1f-4043-470b-827e-6382ee35b8b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:21.225148 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:21.225115 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:21.225341 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:21.225259 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:22.226462 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:22.226424 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:22.226931 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:22.226567 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:22.226931 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:22.226654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:22.226931 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:22.226802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:23.226009 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:23.225636 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:23.226009 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:23.225737 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:24.225271 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:24.225234 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:24.225710 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.225378 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:24.225710 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:24.225427 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:24.225710 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.225556 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:24.843385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:24.843339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:24.843567 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:24.843410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:24.843567 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.843553 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:24.843679 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.843578 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:24.843679 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.843615 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret podName:8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec nodeName:}" failed. No retries permitted until 2026-07-03 15:13:32.843595159 +0000 UTC m=+17.076028036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret") pod "global-pull-secret-syncer-c29qj" (UID: "8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec") : object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:24.843679 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.843643 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs podName:6eb9e801-5990-47ae-964f-6f8e509cffbf nodeName:}" failed. No retries permitted until 2026-07-03 15:13:32.843623895 +0000 UTC m=+17.076056768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs") pod "network-metrics-daemon-rjszq" (UID: "6eb9e801-5990-47ae-964f-6f8e509cffbf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:24.944891 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:24.944224 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:24.944891 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.944458 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 15:13:24.944891 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.944478 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 15:13:24.944891 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.944491 2572 projected.go:196] Error preparing data for projected volume kube-api-access-2bq8p for pod openshift-network-diagnostics/network-check-target-5pbkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:24.944891 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:24.944552 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p podName:abe62e1f-4043-470b-827e-6382ee35b8b7 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:32.944533763 +0000 UTC m=+17.176966639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bq8p" (UniqueName: "kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p") pod "network-check-target-5pbkd" (UID: "abe62e1f-4043-470b-827e-6382ee35b8b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:25.225352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:25.225269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:25.225776 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:25.225412 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:26.226514 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:26.226427 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:26.226514 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:26.226467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:26.227025 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:26.226578 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:26.227025 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:26.226718 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:27.225158 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:27.225103 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:27.225348 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:27.225241 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:28.224999 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:28.224944 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:28.224999 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:28.224999 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:28.225501 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:28.225096 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:28.225501 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:28.225232 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:29.224963 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:29.224878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:29.225138 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:29.225019 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:30.224962 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:30.224919 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:30.225137 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:30.225061 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:30.225137 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:30.225124 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:30.225480 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:30.225230 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:31.225345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:31.225308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:31.225705 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:31.225432 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:32.225046 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:32.225013 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:32.225275 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:32.225176 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:32.225275 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:32.225230 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:32.225391 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:32.225345 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:32.910342 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:32.910305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:32.910538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:32.910362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:32.910538 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:32.910486 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:32.910538 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:32.910510 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:32.910538 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:32.910539 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret podName:8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec nodeName:}" failed. No retries permitted until 2026-07-03 15:13:48.910524475 +0000 UTC m=+33.142957343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret") pod "global-pull-secret-syncer-c29qj" (UID: "8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec") : object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:32.910729 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:32.910574 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs podName:6eb9e801-5990-47ae-964f-6f8e509cffbf nodeName:}" failed. No retries permitted until 2026-07-03 15:13:48.910556241 +0000 UTC m=+33.142989113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs") pod "network-metrics-daemon-rjszq" (UID: "6eb9e801-5990-47ae-964f-6f8e509cffbf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:33.011270 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:33.011237 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:33.011470 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:33.011402 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 15:13:33.011470 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:33.011430 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 15:13:33.011470 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:33.011447 2572 projected.go:196] Error preparing data for projected volume kube-api-access-2bq8p for pod openshift-network-diagnostics/network-check-target-5pbkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:33.011576 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:33.011504 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p podName:abe62e1f-4043-470b-827e-6382ee35b8b7 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:49.011489925 +0000 UTC m=+33.243922793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bq8p" (UniqueName: "kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p") pod "network-check-target-5pbkd" (UID: "abe62e1f-4043-470b-827e-6382ee35b8b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:33.225284 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:33.225219 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:33.225426 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:33.225321 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:34.225076 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:34.225022 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:34.225267 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:34.225022 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:34.225267 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:34.225150 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:34.225267 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:34.225256 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:35.225030 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:35.224982 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:35.225470 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:35.225135 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:36.225980 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:36.225936 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:36.226404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:36.226070 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:36.226404 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:36.226125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:36.226404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:36.226315 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:37.224727 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:37.224688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:37.224885 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:37.224808 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:38.226137 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.225805 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:38.226778 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.225838 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:38.226778 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:38.226236 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:38.226778 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:38.226308 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:38.459055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.458946 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxxq5" event={"ID":"220c038d-e52f-47b6-ab0a-3782911b6f83","Type":"ContainerStarted","Data":"324abba83ec3fbde3b1611a2cb098a5908e9ff9403b8b640edb0c3fea407a7a4"} Jul 03 15:13:38.460324 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.460298 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lct2j" event={"ID":"b2dc4ba3-f734-4053-acee-fed33f36b791","Type":"ContainerStarted","Data":"4007c44e8561393f2b1f416cfad06e6437775b31e4885e15ce4646fd3787f754"} Jul 03 15:13:38.461676 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.461651 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" event={"ID":"fd94fa95-3ca6-469b-93fd-3aae5929e74d","Type":"ContainerStarted","Data":"2a0b9c55b6fb274e178016ce9419fe3f528dc4e15b7cc1c043e395e7bdb62b93"} Jul 03 15:13:38.462953 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.462933 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v62c6" event={"ID":"03c15373-89c7-4c47-8c01-0c6498dedce5","Type":"ContainerStarted","Data":"6b2af8545aa68cdd41e6a2dfed2beaf8d0310f08819f8b7a62b3f7747e036882"} Jul 03 15:13:38.464279 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.464260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq5dz" event={"ID":"82d6aa1f-2b29-4328-83e5-2e961402a35f","Type":"ContainerStarted","Data":"fecd3ce1bebb34bc8039b7a4858d4cba910e7567cb89de6ac4e585265c564907"} Jul 03 15:13:38.465646 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.465626 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e33c5cd-6688-4a91-9744-a36058fdb15f" containerID="27ed754a0d188fc2cb5f1a0a1a5503aba8506e6e1d86c4a094379685649319b3" exitCode=0 Jul 03 15:13:38.465725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.465702 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerDied","Data":"27ed754a0d188fc2cb5f1a0a1a5503aba8506e6e1d86c4a094379685649319b3"} Jul 03 15:13:38.469193 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.469169 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" event={"ID":"c80039b1-5090-4dc1-a555-057c8ddb4337","Type":"ContainerStarted","Data":"bbc8cd40f786e7ea985c960636423ccbdb2fd1c004bc817f677dede6d2fe22e3"} Jul 03 15:13:38.471817 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.471800 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:13:38.472156 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.472140 2572 generic.go:356] "Generic (PLEG): container finished" podID="b762f10c-f840-4d77-9597-e77cc779b7f1" containerID="3b5cc4c2ff75dded31eccedbec899bee8729140a806a056e1c747552182b0a09" exitCode=1 Jul 03 15:13:38.472225 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.472196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"5fa60992c7c1e03aaaf0b1819415073e42b40e8d74f0705de7cee3ac51e70703"} Jul 03 15:13:38.472271 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.472240 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"4659648e96a4cf5c3aef3e72aed1832e48923844a6cf710a485f6c05f3b6740b"} Jul 03 15:13:38.472306 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.472274 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"8de6f576ed12ac2f238696acfb5451760e6e79fb5963cc2a82a69dba78403b14"} Jul 03 15:13:38.472345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.472326 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"3b3b17c19705fd01dc2fba06e5c5d11be0b7930d0c4d022aafd0ec07fca8dfea"} Jul 03 15:13:38.472376 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.472360 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerDied","Data":"3b5cc4c2ff75dded31eccedbec899bee8729140a806a056e1c747552182b0a09"} Jul 03 15:13:38.472425 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.472391 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"15b1f43a3c3e338589dd1bdc185ee7eafe0ed8baee3d98935a5af096cd50f031"} Jul 03 15:13:38.473745 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.473726 2572 generic.go:356] "Generic (PLEG): container finished" podID="c7e9af105ef1fd68636e93b862a9203b" containerID="3a96c6484606e6dae92d51f8f91513262bd9cad7f66bacc152196607ecc7163e" exitCode=0 Jul 03 15:13:38.473833 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.473791 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" event={"ID":"c7e9af105ef1fd68636e93b862a9203b","Type":"ContainerDied","Data":"3a96c6484606e6dae92d51f8f91513262bd9cad7f66bacc152196607ecc7163e"} Jul 03 15:13:38.475598 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.475570 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" event={"ID":"79a3a725957a215de175bf549eccdfcf","Type":"ContainerStarted","Data":"505b7f2dd582ca73100b152bf31c8606eeeb3e2ff1eebbdc537dce5ea1d3c065"} Jul 03 15:13:38.478624 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.478589 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gxxq5" podStartSLOduration=2.408953558 podStartE2EDuration="22.47857945s" totalImagesPullingTime="20.069625892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:38.478286205 +0000 UTC m=+22.710719094" watchObservedRunningTime="2026-07-03 15:13:38.47857945 +0000 UTC m=+22.711012339" Jul 03 15:13:38.518455 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.518387 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lq5dz" podStartSLOduration=2.756973504 podStartE2EDuration="22.518368679s" totalImagesPullingTime="19.761395175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:38.517678153 +0000 UTC m=+22.750111037" watchObservedRunningTime="2026-07-03 15:13:38.518368679 +0000 UTC m=+22.750801569" Jul 03 15:13:38.550647 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.550389 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" podStartSLOduration=21.550371361 podStartE2EDuration="21.550371361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:38.549256373 +0000 UTC m=+22.781689262" watchObservedRunningTime="2026-07-03 15:13:38.550371361 +0000 UTC m=+22.782804251" Jul 03 15:13:38.565351 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.565296 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lct2j" podStartSLOduration=2.705367223 podStartE2EDuration="22.565274973s" totalImagesPullingTime="19.85990775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:38.565003004 +0000 UTC m=+22.797435890" watchObservedRunningTime="2026-07-03 15:13:38.565274973 +0000 UTC m=+22.797707868" Jul 03 15:13:38.583376 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.583328 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v62c6" podStartSLOduration=2.745365189 podStartE2EDuration="22.583307772s" totalImagesPullingTime="19.837942583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:38.582938961 +0000 UTC m=+22.815371850" watchObservedRunningTime="2026-07-03 15:13:38.583307772 +0000 UTC m=+22.815740664" Jul 03 15:13:38.601608 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:38.601566 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5sp5v" podStartSLOduration=2.735398145 podStartE2EDuration="22.601548969s" totalImagesPullingTime="19.866150824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:38.601191157 +0000 UTC m=+22.833624048" watchObservedRunningTime="2026-07-03 15:13:38.601548969 +0000 UTC m=+22.833981859" Jul 03 15:13:39.225584 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:39.225545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:39.225810 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:39.225650 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:39.469862 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:39.469807 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 03 15:13:39.479931 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:39.479864 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" event={"ID":"c7e9af105ef1fd68636e93b862a9203b","Type":"ContainerStarted","Data":"8a9a240171ca93bb39fddfbb4d52b3a9076f9127d88c7ce6892a881356ceeacd"} Jul 03 15:13:39.481384 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:39.481355 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j4crr" event={"ID":"361fddc6-1cd2-433b-8ced-6e459e1f3094","Type":"ContainerStarted","Data":"31536c0a91c0b192144420fe9eaaf9bb8ade53dba7565bafbe5c431ad6f74d9c"} Jul 03 15:13:39.483439 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:39.483398 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" event={"ID":"fd94fa95-3ca6-469b-93fd-3aae5929e74d","Type":"ContainerStarted","Data":"26ff24a996ba6bb61335f32e702ba4f6685d1ba693fa3230a8896ac23dae8880"} Jul 03 15:13:39.494503 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:39.494467 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" podStartSLOduration=22.494454868 podStartE2EDuration="22.494454868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:39.494195962 +0000 UTC m=+23.726628852" watchObservedRunningTime="2026-07-03 15:13:39.494454868 +0000 UTC m=+23.726887757" Jul 03 15:13:39.514382 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:39.514341 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j4crr" podStartSLOduration=3.74887462 podStartE2EDuration="23.51432928s" totalImagesPullingTime="19.76545466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:39.512445733 +0000 UTC m=+23.744878622" watchObservedRunningTime="2026-07-03 15:13:39.51432928 +0000 UTC m=+23.746762169" Jul 03 15:13:40.134218 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:40.134032 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-03T15:13:39.469823689Z","UUID":"071e1177-aed9-45c9-9b5f-a66e5954f532","Handler":null,"Name":"","Endpoint":""} Jul 03 15:13:40.135703 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:40.135664 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 03 15:13:40.135848 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:40.135733 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 03 15:13:40.225712 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:40.225045 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:40.225712 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:40.225183 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:40.225712 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:40.225481 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:40.225712 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:40.225574 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:40.490527 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:40.490451 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:13:40.491217 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:40.491187 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"844b7abcc9141f4e4498f4a1286c40ed3cdcaf7a2e8a99fc390af3b8f6a72fd9"} Jul 03 15:13:41.225687 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:41.225435 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:41.225833 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:41.225769 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:41.495928 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:41.495843 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" event={"ID":"fd94fa95-3ca6-469b-93fd-3aae5929e74d","Type":"ContainerStarted","Data":"497d257f4897f5b921369e3a4ead15cc9e80f11f1907b5269c9a9428805fb55d"} Jul 03 15:13:41.513432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:41.513371 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c6c" podStartSLOduration=3.027481371 podStartE2EDuration="25.513352936s" totalImagesPullingTime="22.485871565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:41.512070572 +0000 UTC m=+25.744503460" watchObservedRunningTime="2026-07-03 15:13:41.513352936 +0000 UTC m=+25.745785826" Jul 03 15:13:42.225621 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.225584 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:42.225621 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.225608 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:42.225863 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:42.225719 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:42.225863 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:42.225815 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:42.377263 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.377237 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:42.377785 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.377769 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:42.502061 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.502040 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:13:42.502731 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.502432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"cc1a7b724d9e59add50abf4d1c2ca194aaf2891bb75685e274f81a9c5a0d493e"} Jul 03 15:13:42.502857 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.502840 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:42.502936 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.502879 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:42.503011 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.502939 2572 scope.go:117] "RemoveContainer" containerID="3b5cc4c2ff75dded31eccedbec899bee8729140a806a056e1c747552182b0a09" Jul 03 15:13:42.503383 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:42.503369 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v62c6" Jul 03 15:13:43.225524 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.225332 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:43.225686 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:43.225617 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:43.508892 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.508792 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:13:43.509505 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.509216 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" event={"ID":"b762f10c-f840-4d77-9597-e77cc779b7f1","Type":"ContainerStarted","Data":"0e5da147dfff5015f6f6d55da13bd8403db24cc14fefde5ce4e2b54a7cd51f2b"} Jul 03 15:13:43.509620 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.509605 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:43.509686 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.509675 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:43.527544 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.527514 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:43.530161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.528417 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:13:43.539323 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:43.539265 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" podStartSLOduration=7.612569292 podStartE2EDuration="27.539248487s" totalImagesPullingTime="19.926679195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:43.537380943 +0000 UTC m=+27.769813873" watchObservedRunningTime="2026-07-03 15:13:43.539248487 +0000 UTC m=+27.771681375" Jul 03 15:13:44.225494 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.225412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:44.225637 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:44.225553 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:44.225751 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.225412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:44.225863 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:44.225840 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:44.660266 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.660234 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c29qj"] Jul 03 15:13:44.660713 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.660354 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:44.660713 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:44.660463 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:44.669683 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.669513 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5pbkd"] Jul 03 15:13:44.669683 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.669644 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:44.669874 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:44.669761 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:44.670196 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.670165 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rjszq"] Jul 03 15:13:44.670303 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:44.670261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:44.670370 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:44.670354 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:45.514602 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:45.514569 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e33c5cd-6688-4a91-9744-a36058fdb15f" containerID="9ed6d1fd469025439da98f78968bfc42353d97aa75555d229f0f9b1dbef888a1" exitCode=0 Jul 03 15:13:45.514736 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:45.514665 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerDied","Data":"9ed6d1fd469025439da98f78968bfc42353d97aa75555d229f0f9b1dbef888a1"} Jul 03 15:13:46.226334 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:46.226062 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:46.226334 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:46.226115 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:46.226842 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:46.226341 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:46.226842 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:46.226433 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:46.226842 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:46.226150 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:46.226842 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:46.226521 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:46.518551 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:46.518511 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e33c5cd-6688-4a91-9744-a36058fdb15f" containerID="17f89f66b31200b861a9e7c3a2c4b768cde543ae9e25de8ce15e8a99e2dd7599" exitCode=0 Jul 03 15:13:46.518680 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:46.518568 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerDied","Data":"17f89f66b31200b861a9e7c3a2c4b768cde543ae9e25de8ce15e8a99e2dd7599"} Jul 03 15:13:47.522982 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:47.522896 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e33c5cd-6688-4a91-9744-a36058fdb15f" containerID="c8cc56cbdc4fff189578ca8049a4c0bbd3278a5d7588d0e0faa933035dcd9ed7" exitCode=0 Jul 03 15:13:47.522982 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:47.522958 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerDied","Data":"c8cc56cbdc4fff189578ca8049a4c0bbd3278a5d7588d0e0faa933035dcd9ed7"} Jul 03 15:13:48.227652 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:48.227624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:48.227814 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:48.227624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:48.227814 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:48.227724 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5pbkd" podUID="abe62e1f-4043-470b-827e-6382ee35b8b7" Jul 03 15:13:48.227893 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:48.227809 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rjszq" podUID="6eb9e801-5990-47ae-964f-6f8e509cffbf" Jul 03 15:13:48.227893 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:48.227624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:48.227960 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:48.227920 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c29qj" podUID="8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec" Jul 03 15:13:48.930919 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:48.930881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:48.931445 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:48.930953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:48.931445 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:48.931083 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:48.931445 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:48.931085 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:48.931445 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:48.931150 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret podName:8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec nodeName:}" failed. No retries permitted until 2026-07-03 15:14:20.931132455 +0000 UTC m=+65.163565325 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret") pod "global-pull-secret-syncer-c29qj" (UID: "8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec") : object "kube-system"/"original-pull-secret" not registered Jul 03 15:13:48.931445 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:48.931171 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs podName:6eb9e801-5990-47ae-964f-6f8e509cffbf nodeName:}" failed. No retries permitted until 2026-07-03 15:14:20.93116071 +0000 UTC m=+65.163593581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs") pod "network-metrics-daemon-rjszq" (UID: "6eb9e801-5990-47ae-964f-6f8e509cffbf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 15:13:49.032168 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:49.032127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:49.032332 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:49.032266 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 15:13:49.032332 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:49.032294 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 15:13:49.032332 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:49.032305 2572 projected.go:196] Error preparing data for projected volume kube-api-access-2bq8p for pod openshift-network-diagnostics/network-check-target-5pbkd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:49.032450 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:49.032366 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p podName:abe62e1f-4043-470b-827e-6382ee35b8b7 nodeName:}" failed. No retries permitted until 2026-07-03 15:14:21.032352899 +0000 UTC m=+65.264785769 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bq8p" (UniqueName: "kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p") pod "network-check-target-5pbkd" (UID: "abe62e1f-4043-470b-827e-6382ee35b8b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 15:13:50.066304 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.066273 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeReady" Jul 03 15:13:50.066772 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.066431 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 03 15:13:50.102781 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.102743 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-546bfb8bc9-2mttx"] Jul 03 15:13:50.105619 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.105595 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.108424 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.108400 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 03 15:13:50.108565 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.108540 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 03 15:13:50.108695 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.108672 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 03 15:13:50.108757 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.108701 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ppp7b\"" Jul 03 15:13:50.113719 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.113687 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kpqg8"] Jul 03 15:13:50.114692 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.114665 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 03 15:13:50.117149 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.117132 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qmjkm"] Jul 03 15:13:50.117230 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.117199 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.120291 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.120262 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546bfb8bc9-2mttx"] Jul 03 15:13:50.120399 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.120284 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 03 15:13:50.120399 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.120357 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.120633 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.120612 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 03 15:13:50.120845 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.120827 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 03 15:13:50.120950 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.120851 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 03 15:13:50.121055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.120874 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2xkrx\"" Jul 03 15:13:50.122982 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.122893 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 03 15:13:50.123140 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.123111 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 03 15:13:50.123209 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.123177 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dft5m\"" Jul 03 15:13:50.123700 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.123685 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 03 15:13:50.131170 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.131142 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kpqg8"] Jul 03 15:13:50.135973 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.135950 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmjkm"] Jul 03 15:13:50.138799 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.138775 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2wrgx"] Jul 03 15:13:50.142328 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.142307 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.145057 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.145037 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w99m5\"" Jul 03 15:13:50.145185 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.145166 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 03 15:13:50.145326 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.145301 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 03 15:13:50.151719 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.151696 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2wrgx"] Jul 03 15:13:50.225250 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.225215 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:13:50.225426 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.225214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:13:50.225426 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.225361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:13:50.228031 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.228010 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 15:13:50.228031 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.228011 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 03 15:13:50.228234 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.228066 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 15:13:50.228234 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.228097 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9nf84\"" Jul 03 15:13:50.228234 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.228122 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 03 15:13:50.228412 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.228398 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cq8c8\"" Jul 03 15:13:50.239790 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.239767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-data-volume\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.239894 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.239802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz76d\" (UniqueName: \"kubernetes.io/projected/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-api-access-hz76d\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.239894 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.239852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.239894 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.239876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2ba0c68-6bca-41e6-9915-077e0dd35a08-ca-trust-extracted\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.239915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-config-volume\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.240055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.239939 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-certificates\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.239963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-bound-sa-token\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240200 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkqp\" (UniqueName: \"kubernetes.io/projected/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-kube-api-access-8pkqp\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.240200 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.240200 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240122 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-image-registry-private-configuration\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240200 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240150 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzd8k\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-kube-api-access-wzd8k\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-tmp-dir\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.240385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-trusted-ca\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q7r\" (UniqueName: \"kubernetes.io/projected/a97e8698-dc26-4b82-9bfb-60c7b176c487-kube-api-access-q8q7r\") pod \"ingress-canary-qmjkm\" (UID: \"a97e8698-dc26-4b82-9bfb-60c7b176c487\") " pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.240385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240298 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-installation-pull-secrets\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.240385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a97e8698-dc26-4b82-9bfb-60c7b176c487-cert\") pod \"ingress-canary-qmjkm\" (UID: \"a97e8698-dc26-4b82-9bfb-60c7b176c487\") " pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.240385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240381 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-crio-socket\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.240622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.240430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.341285 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2ba0c68-6bca-41e6-9915-077e0dd35a08-ca-trust-extracted\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-config-volume\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-certificates\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341358 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-bound-sa-token\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341387 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkqp\" (UniqueName: \"kubernetes.io/projected/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-kube-api-access-8pkqp\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.341414 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-image-registry-private-configuration\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341461 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzd8k\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-kube-api-access-wzd8k\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.341487 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls podName:0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:50.841467592 +0000 UTC m=+35.073900458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls") pod "dns-default-2wrgx" (UID: "0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83") : secret "dns-default-metrics-tls" not found Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.341515 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-tmp-dir\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341583 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-trusted-ca\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.341601 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:50.841581803 +0000 UTC m=+35.074014684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : configmap "kube-rbac-proxy" not found Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q7r\" (UniqueName: \"kubernetes.io/projected/a97e8698-dc26-4b82-9bfb-60c7b176c487-kube-api-access-q8q7r\") pod \"ingress-canary-qmjkm\" (UID: \"a97e8698-dc26-4b82-9bfb-60c7b176c487\") " pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341728 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-installation-pull-secrets\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a97e8698-dc26-4b82-9bfb-60c7b176c487-cert\") pod \"ingress-canary-qmjkm\" (UID: \"a97e8698-dc26-4b82-9bfb-60c7b176c487\") " pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341796 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2ba0c68-6bca-41e6-9915-077e0dd35a08-ca-trust-extracted\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-crio-socket\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.341873 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.341900 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546bfb8bc9-2mttx: secret "image-registry-tls" not found Jul 03 15:13:50.341907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.341900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.341963 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls podName:c2ba0c68-6bca-41e6-9915-077e0dd35a08 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:50.841940465 +0000 UTC m=+35.074373333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls") pod "image-registry-546bfb8bc9-2mttx" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08") : secret "image-registry-tls" not found Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342014 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-config-volume\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-certificates\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-tmp-dir\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.342020 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342133 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-data-volume\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz76d\" (UniqueName: \"kubernetes.io/projected/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-api-access-hz76d\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.342188 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:50.842172604 +0000 UTC m=+35.074605484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : secret "insights-runtime-extractor-tls" not found Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342250 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-crio-socket\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.342510 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342419 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-data-volume\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.342853 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.342597 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-trusted-ca\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.345815 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.345792 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-image-registry-private-configuration\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.345815 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.345800 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a97e8698-dc26-4b82-9bfb-60c7b176c487-cert\") pod \"ingress-canary-qmjkm\" (UID: \"a97e8698-dc26-4b82-9bfb-60c7b176c487\") " pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.345964 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.345851 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-installation-pull-secrets\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.353128 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.352823 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkqp\" (UniqueName: \"kubernetes.io/projected/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-kube-api-access-8pkqp\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.353128 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.353084 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzd8k\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-kube-api-access-wzd8k\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.353327 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.353217 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q7r\" (UniqueName: \"kubernetes.io/projected/a97e8698-dc26-4b82-9bfb-60c7b176c487-kube-api-access-q8q7r\") pod \"ingress-canary-qmjkm\" (UID: \"a97e8698-dc26-4b82-9bfb-60c7b176c487\") " pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.353641 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.353616 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-bound-sa-token\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.354624 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.354601 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz76d\" (UniqueName: \"kubernetes.io/projected/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-api-access-hz76d\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.440472 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.440434 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmjkm" Jul 03 15:13:50.600610 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.600357 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmjkm"] Jul 03 15:13:50.605398 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:50.605374 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97e8698_dc26_4b82_9bfb_60c7b176c487.slice/crio-7dc24b15cfeb63b2ee9b5b04f2c564196ba2169d179d8047202ff45401337918 WatchSource:0}: Error finding container 7dc24b15cfeb63b2ee9b5b04f2c564196ba2169d179d8047202ff45401337918: Status 404 returned error can't find the container with id 7dc24b15cfeb63b2ee9b5b04f2c564196ba2169d179d8047202ff45401337918 Jul 03 15:13:50.845847 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.845814 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:50.846074 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.845866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.846074 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.845930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:50.846074 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:50.845973 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:50.846074 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846024 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 15:13:50.846074 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846050 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546bfb8bc9-2mttx: secret "image-registry-tls" not found Jul 03 15:13:50.846074 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846054 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 15:13:50.846373 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846054 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 15:13:50.846373 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846102 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 15:13:50.846373 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846111 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls podName:c2ba0c68-6bca-41e6-9915-077e0dd35a08 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:51.846092014 +0000 UTC m=+36.078524900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls") pod "image-registry-546bfb8bc9-2mttx" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08") : secret "image-registry-tls" not found Jul 03 15:13:50.846373 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846139 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:51.846118885 +0000 UTC m=+36.078551775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : configmap "kube-rbac-proxy" not found Jul 03 15:13:50.846373 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846162 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:51.846148734 +0000 UTC m=+36.078581602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : secret "insights-runtime-extractor-tls" not found Jul 03 15:13:50.846373 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:50.846182 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls podName:0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:51.84617411 +0000 UTC m=+36.078606978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls") pod "dns-default-2wrgx" (UID: "0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83") : secret "dns-default-metrics-tls" not found Jul 03 15:13:51.366558 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.366520 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-ffl7g"] Jul 03 15:13:51.403516 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.403485 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-ffl7g"] Jul 03 15:13:51.403685 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.403631 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.406412 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.406388 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 03 15:13:51.407552 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.407524 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 03 15:13:51.407674 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.407569 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8cmmw\"" Jul 03 15:13:51.407674 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.407586 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 03 15:13:51.407674 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.407606 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 03 15:13:51.535575 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.535535 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmjkm" event={"ID":"a97e8698-dc26-4b82-9bfb-60c7b176c487","Type":"ContainerStarted","Data":"7dc24b15cfeb63b2ee9b5b04f2c564196ba2169d179d8047202ff45401337918"} Jul 03 15:13:51.552114 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.552084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/18180af7-8247-42bb-9529-7fb41c6e9d06-signing-cabundle\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.552307 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.552181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/18180af7-8247-42bb-9529-7fb41c6e9d06-signing-key\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.552307 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.552238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2t79\" (UniqueName: \"kubernetes.io/projected/18180af7-8247-42bb-9529-7fb41c6e9d06-kube-api-access-k2t79\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.624663 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.624590 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq5dz_82d6aa1f-2b29-4328-83e5-2e961402a35f/dns-node-resolver/0.log" Jul 03 15:13:51.653013 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.652965 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/18180af7-8247-42bb-9529-7fb41c6e9d06-signing-cabundle\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.653192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.653073 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/18180af7-8247-42bb-9529-7fb41c6e9d06-signing-key\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.653192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.653138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2t79\" (UniqueName: \"kubernetes.io/projected/18180af7-8247-42bb-9529-7fb41c6e9d06-kube-api-access-k2t79\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.653792 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.653767 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/18180af7-8247-42bb-9529-7fb41c6e9d06-signing-cabundle\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.656799 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.656775 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/18180af7-8247-42bb-9529-7fb41c6e9d06-signing-key\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.661972 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.661947 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2t79\" (UniqueName: \"kubernetes.io/projected/18180af7-8247-42bb-9529-7fb41c6e9d06-kube-api-access-k2t79\") pod \"service-ca-758f599c5-ffl7g\" (UID: \"18180af7-8247-42bb-9529-7fb41c6e9d06\") " pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.714441 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.714406 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-ffl7g" Jul 03 15:13:51.855682 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.855638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:51.855873 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.855717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:51.855873 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.855749 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:51.855873 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.855794 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856101 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856235 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls podName:0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:53.856213648 +0000 UTC m=+38.088646517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls") pod "dns-default-2wrgx" (UID: "0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83") : secret "dns-default-metrics-tls" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856263 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:53.856252808 +0000 UTC m=+38.088685676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : secret "insights-runtime-extractor-tls" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856264 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856312 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:53.856295161 +0000 UTC m=+38.088728040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : configmap "kube-rbac-proxy" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:51.856727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856858 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856870 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546bfb8bc9-2mttx: secret "image-registry-tls" not found Jul 03 15:13:51.859404 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:51.856910 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls podName:c2ba0c68-6bca-41e6-9915-077e0dd35a08 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:53.856893279 +0000 UTC m=+38.089326145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls") pod "image-registry-546bfb8bc9-2mttx" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08") : secret "image-registry-tls" not found Jul 03 15:13:52.313480 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:52.313448 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-ffl7g"] Jul 03 15:13:52.489935 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:13:52.489842 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18180af7_8247_42bb_9529_7fb41c6e9d06.slice/crio-05b7a5b54a1b6b88648b02f502dd53e4fafaee7ee323ca0faaedcaa24cf09403 WatchSource:0}: Error finding container 05b7a5b54a1b6b88648b02f502dd53e4fafaee7ee323ca0faaedcaa24cf09403: Status 404 returned error can't find the container with id 05b7a5b54a1b6b88648b02f502dd53e4fafaee7ee323ca0faaedcaa24cf09403 Jul 03 15:13:52.539352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:52.539320 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-ffl7g" event={"ID":"18180af7-8247-42bb-9529-7fb41c6e9d06","Type":"ContainerStarted","Data":"05b7a5b54a1b6b88648b02f502dd53e4fafaee7ee323ca0faaedcaa24cf09403"} Jul 03 15:13:52.998172 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:52.998142 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lct2j_b2dc4ba3-f734-4053-acee-fed33f36b791/node-ca/0.log" Jul 03 15:13:53.548295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:53.548169 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e33c5cd-6688-4a91-9744-a36058fdb15f" containerID="ccfa5b13622913c961819f988c88ecf22ab395c08a7eefa04cc923da4ac30c2a" exitCode=0 Jul 03 15:13:53.548295 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:53.548240 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerDied","Data":"ccfa5b13622913c961819f988c88ecf22ab395c08a7eefa04cc923da4ac30c2a"} Jul 03 15:13:53.871483 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:53.871267 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:53.871662 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:53.871523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:53.871662 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:53.871382 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 15:13:53.871662 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:53.871561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:53.871662 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:53.871617 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:57.871596069 +0000 UTC m=+42.104028938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : secret "insights-runtime-extractor-tls" not found Jul 03 15:13:53.871880 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:53.871668 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 15:13:53.871880 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:53.871678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:53.871880 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:53.871724 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls podName:0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:57.871706101 +0000 UTC m=+42.104138981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls") pod "dns-default-2wrgx" (UID: "0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83") : secret "dns-default-metrics-tls" not found Jul 03 15:13:53.871880 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:53.871775 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 15:13:53.871880 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:53.871789 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546bfb8bc9-2mttx: secret "image-registry-tls" not found Jul 03 15:13:53.871880 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:53.871831 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls podName:c2ba0c68-6bca-41e6-9915-077e0dd35a08 nodeName:}" failed. No retries permitted until 2026-07-03 15:13:57.871818716 +0000 UTC m=+42.104251603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls") pod "image-registry-546bfb8bc9-2mttx" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08") : secret "image-registry-tls" not found Jul 03 15:13:53.872126 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:53.872108 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:55.554671 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:55.554584 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-ffl7g" event={"ID":"18180af7-8247-42bb-9529-7fb41c6e9d06","Type":"ContainerStarted","Data":"3ece7f1b962463a9a577c2c7ee84d0c91f655cece8bbf8c7aa942aee41c27941"} Jul 03 15:13:55.557690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:55.557665 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e33c5cd-6688-4a91-9744-a36058fdb15f" containerID="46bef2ffd6ac33b505179e1a7a78aab2ab791b420b0980e27d361ae10e41fe46" exitCode=0 Jul 03 15:13:55.557811 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:55.557753 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerDied","Data":"46bef2ffd6ac33b505179e1a7a78aab2ab791b420b0980e27d361ae10e41fe46"} Jul 03 15:13:55.559187 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:55.559161 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmjkm" event={"ID":"a97e8698-dc26-4b82-9bfb-60c7b176c487","Type":"ContainerStarted","Data":"b0588ef60dd52634778b9d3a4ab4c54902ff89853fb85345ba82d40c0afb3408"} Jul 03 15:13:55.575483 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:55.575433 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-ffl7g" podStartSLOduration=1.941243434 podStartE2EDuration="4.575411883s" totalImagesPullingTime="2.634168449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:55.574119001 +0000 UTC m=+39.806551891" watchObservedRunningTime="2026-07-03 15:13:55.575411883 +0000 UTC m=+39.807844767" Jul 03 15:13:55.613809 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:55.613753 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qmjkm" podStartSLOduration=1.082410128 podStartE2EDuration="5.613735426s" totalImagesPullingTime="4.531325298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:55.6131929 +0000 UTC m=+39.845625792" watchObservedRunningTime="2026-07-03 15:13:55.613735426 +0000 UTC m=+39.846168315" Jul 03 15:13:56.567096 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:56.567025 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk58z" event={"ID":"5e33c5cd-6688-4a91-9744-a36058fdb15f","Type":"ContainerStarted","Data":"9e8128ab8854771643206738e98d52666fff87890be76d35f8c0bac259d23f8f"} Jul 03 15:13:56.591849 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:56.591778 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jk58z" podStartSLOduration=7.990255548 podStartE2EDuration="40.591757459s" totalImagesPullingTime="32.601501911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:13:56.590132096 +0000 UTC m=+40.822564986" watchObservedRunningTime="2026-07-03 15:13:56.591757459 +0000 UTC m=+40.824190350" Jul 03 15:13:57.908916 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:57.908875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:13:57.908916 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:57.908924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:13:57.909031 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:57.909057 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:57.909080 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546bfb8bc9-2mttx: secret "image-registry-tls" not found Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:57.909099 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:57.909127 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:57.909133 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls podName:c2ba0c68-6bca-41e6-9915-077e0dd35a08 nodeName:}" failed. No retries permitted until 2026-07-03 15:14:05.909117163 +0000 UTC m=+50.141550030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls") pod "image-registry-546bfb8bc9-2mttx" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08") : secret "image-registry-tls" not found Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:57.909153 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls podName:8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8 nodeName:}" failed. No retries permitted until 2026-07-03 15:14:05.909144286 +0000 UTC m=+50.141577159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kpqg8" (UID: "8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8") : secret "insights-runtime-extractor-tls" not found Jul 03 15:13:57.909335 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:13:57.909167 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls podName:0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83 nodeName:}" failed. No retries permitted until 2026-07-03 15:14:05.909159527 +0000 UTC m=+50.141592393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls") pod "dns-default-2wrgx" (UID: "0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83") : secret "dns-default-metrics-tls" not found Jul 03 15:14:05.980100 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:05.980063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:14:05.980559 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:05.980116 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:14:05.980559 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:05.980167 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:14:05.983693 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:05.983668 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kpqg8\" (UID: \"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8\") " pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:14:05.983827 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:05.983722 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83-metrics-tls\") pod \"dns-default-2wrgx\" (UID: \"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83\") " pod="openshift-dns/dns-default-2wrgx" Jul 03 15:14:05.983827 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:05.983769 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"image-registry-546bfb8bc9-2mttx\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:14:06.018144 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.018109 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:14:06.031274 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.031245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kpqg8" Jul 03 15:14:06.052219 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.052185 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2wrgx" Jul 03 15:14:06.194844 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:06.194803 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ba0c68_6bca_41e6_9915_077e0dd35a08.slice/crio-d565d45a537bd8f8d9b8e926b5bf26aae1762a8fb62e152ee88323eae0df3357 WatchSource:0}: Error finding container d565d45a537bd8f8d9b8e926b5bf26aae1762a8fb62e152ee88323eae0df3357: Status 404 returned error can't find the container with id d565d45a537bd8f8d9b8e926b5bf26aae1762a8fb62e152ee88323eae0df3357 Jul 03 15:14:06.203547 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.203440 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546bfb8bc9-2mttx"] Jul 03 15:14:06.218339 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.218307 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kpqg8"] Jul 03 15:14:06.221071 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:06.221044 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5f109c_b88f_4b1c_aa9f_4d8615abd2c8.slice/crio-0d31ede48200fc59c0e628fd31bf12eb8f8170a69cd38db82be92431bb32f1ba WatchSource:0}: Error finding container 0d31ede48200fc59c0e628fd31bf12eb8f8170a69cd38db82be92431bb32f1ba: Status 404 returned error can't find the container with id 0d31ede48200fc59c0e628fd31bf12eb8f8170a69cd38db82be92431bb32f1ba Jul 03 15:14:06.240714 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.240691 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2wrgx"] Jul 03 15:14:06.243393 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:06.243371 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d91cdbf_9e99_4ebe_adcd_ea3d7adf1b83.slice/crio-f1ed14a316c81cf7a59d48a6d8a1eab8c7051388e7a23b54706140f5dd9c8437 WatchSource:0}: Error finding container f1ed14a316c81cf7a59d48a6d8a1eab8c7051388e7a23b54706140f5dd9c8437: Status 404 returned error can't find the container with id f1ed14a316c81cf7a59d48a6d8a1eab8c7051388e7a23b54706140f5dd9c8437 Jul 03 15:14:06.592124 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.592088 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpqg8" event={"ID":"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8","Type":"ContainerStarted","Data":"0d31ede48200fc59c0e628fd31bf12eb8f8170a69cd38db82be92431bb32f1ba"} Jul 03 15:14:06.593595 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.593570 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" event={"ID":"c2ba0c68-6bca-41e6-9915-077e0dd35a08","Type":"ContainerStarted","Data":"b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b"} Jul 03 15:14:06.593744 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.593610 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" event={"ID":"c2ba0c68-6bca-41e6-9915-077e0dd35a08","Type":"ContainerStarted","Data":"d565d45a537bd8f8d9b8e926b5bf26aae1762a8fb62e152ee88323eae0df3357"} Jul 03 15:14:06.593744 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.593704 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:14:06.594597 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.594572 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2wrgx" event={"ID":"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83","Type":"ContainerStarted","Data":"f1ed14a316c81cf7a59d48a6d8a1eab8c7051388e7a23b54706140f5dd9c8437"} Jul 03 15:14:06.616242 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:06.616192 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" podStartSLOduration=50.61617634 podStartE2EDuration="50.61617634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:06.614838791 +0000 UTC m=+50.847271680" watchObservedRunningTime="2026-07-03 15:14:06.61617634 +0000 UTC m=+50.848609207" Jul 03 15:14:11.611521 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:11.611479 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpqg8" event={"ID":"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8","Type":"ContainerStarted","Data":"41d6b2bd1ee326918ccb7f4a7d11bdf103280256d88b625070be18e6d871ec9e"} Jul 03 15:14:11.613486 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:11.613453 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2wrgx" event={"ID":"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83","Type":"ContainerStarted","Data":"797f5389f5559a2c73113a353fa7055d1844cecf6a2f64e767c0a6999a5df18e"} Jul 03 15:14:11.613614 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:11.613503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2wrgx" event={"ID":"0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83","Type":"ContainerStarted","Data":"8830d2af950b2ba39455b5f2e3f1487367baa815519fb02c8d923402e5f17899"} Jul 03 15:14:11.613614 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:11.613608 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2wrgx" Jul 03 15:14:11.630293 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:11.630246 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2wrgx" podStartSLOduration=16.858391522 podStartE2EDuration="21.630230901s" totalImagesPullingTime="4.771839379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:11.628876533 +0000 UTC m=+55.861309423" watchObservedRunningTime="2026-07-03 15:14:11.630230901 +0000 UTC m=+55.862663789" Jul 03 15:14:12.620063 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.620031 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpqg8" event={"ID":"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8","Type":"ContainerStarted","Data":"695a381ea5d84185b6a2df79fbcdcc3a720e88437541380eb97bc93b36b65e4d"} Jul 03 15:14:12.926117 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.926039 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546bfb8bc9-2mttx"] Jul 03 15:14:12.942007 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.941710 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5"] Jul 03 15:14:12.943828 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.943802 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt"] Jul 03 15:14:12.943977 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.943955 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:12.945857 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.945832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:12.947214 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.946962 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 03 15:14:12.947214 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.947077 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 03 15:14:12.948818 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.948795 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 03 15:14:12.949072 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.949056 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 03 15:14:12.951780 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.951758 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 03 15:14:12.952198 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.952156 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 03 15:14:12.952198 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.952162 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 03 15:14:12.952553 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.952394 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 03 15:14:12.960214 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.960157 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5"] Jul 03 15:14:12.967033 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:12.966970 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt"] Jul 03 15:14:13.035611 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035569 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.035797 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.035797 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/333eb90f-cbae-4ade-9018-9102fed3f9dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.035797 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035718 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgll\" (UniqueName: \"kubernetes.io/projected/333eb90f-cbae-4ade-9018-9102fed3f9dd-kube-api-access-6bgll\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.035797 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333eb90f-cbae-4ade-9018-9102fed3f9dd-tmp\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.035797 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035766 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-hub\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.036062 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035845 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-ca\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.036062 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c2172e-7b87-492d-bc80-5f6146a17b9f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.036062 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.035942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7q5t\" (UniqueName: \"kubernetes.io/projected/15c2172e-7b87-492d-bc80-5f6146a17b9f-kube-api-access-m7q5t\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.042658 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.042626 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-646bbcd79b-9792s"] Jul 03 15:14:13.044604 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.044582 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.070454 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.070424 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646bbcd79b-9792s"] Jul 03 15:14:13.136723 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.136911 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-bound-sa-token\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.136911 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/333eb90f-cbae-4ade-9018-9102fed3f9dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.136911 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgll\" (UniqueName: \"kubernetes.io/projected/333eb90f-cbae-4ade-9018-9102fed3f9dd-kube-api-access-6bgll\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.136911 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-registry-tls\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.136911 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9413fe8e-3dfb-4502-be04-d26b3cae42ba-installation-pull-secrets\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.136911 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333eb90f-cbae-4ade-9018-9102fed3f9dd-tmp\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-hub\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9413fe8e-3dfb-4502-be04-d26b3cae42ba-ca-trust-extracted\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.136970 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-ca\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9413fe8e-3dfb-4502-be04-d26b3cae42ba-image-registry-private-configuration\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c2172e-7b87-492d-bc80-5f6146a17b9f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7q5t\" (UniqueName: \"kubernetes.io/projected/15c2172e-7b87-492d-bc80-5f6146a17b9f-kube-api-access-m7q5t\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137123 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9413fe8e-3dfb-4502-be04-d26b3cae42ba-registry-certificates\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.137165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137160 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9413fe8e-3dfb-4502-be04-d26b3cae42ba-trusted-ca\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.137560 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzczd\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-kube-api-access-zzczd\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.137560 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.137560 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137537 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333eb90f-cbae-4ade-9018-9102fed3f9dd-tmp\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.138027 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.137980 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c2172e-7b87-492d-bc80-5f6146a17b9f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.139688 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.139630 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.139812 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.139763 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/333eb90f-cbae-4ade-9018-9102fed3f9dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.139920 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.139897 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-ca\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.140242 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.140218 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-hub\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.140303 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.140248 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15c2172e-7b87-492d-bc80-5f6146a17b9f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.153615 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.153588 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7q5t\" (UniqueName: \"kubernetes.io/projected/15c2172e-7b87-492d-bc80-5f6146a17b9f-kube-api-access-m7q5t\") pod \"cluster-proxy-proxy-agent-6b8f977655-lrdd5\" (UID: \"15c2172e-7b87-492d-bc80-5f6146a17b9f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.158807 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.158782 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgll\" (UniqueName: \"kubernetes.io/projected/333eb90f-cbae-4ade-9018-9102fed3f9dd-kube-api-access-6bgll\") pod \"klusterlet-addon-workmgr-64b8b6868d-v2fnt\" (UID: \"333eb90f-cbae-4ade-9018-9102fed3f9dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.237871 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.237803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9413fe8e-3dfb-4502-be04-d26b3cae42ba-ca-trust-extracted\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.237871 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.237847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9413fe8e-3dfb-4502-be04-d26b3cae42ba-image-registry-private-configuration\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.237878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9413fe8e-3dfb-4502-be04-d26b3cae42ba-registry-certificates\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.237901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9413fe8e-3dfb-4502-be04-d26b3cae42ba-trusted-ca\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.237918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzczd\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-kube-api-access-zzczd\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.238018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-bound-sa-token\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.238056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-registry-tls\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.238092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9413fe8e-3dfb-4502-be04-d26b3cae42ba-installation-pull-secrets\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238403 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.238266 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9413fe8e-3dfb-4502-be04-d26b3cae42ba-ca-trust-extracted\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.238977 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.238951 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9413fe8e-3dfb-4502-be04-d26b3cae42ba-registry-certificates\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.239274 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.239251 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9413fe8e-3dfb-4502-be04-d26b3cae42ba-trusted-ca\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.240378 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.240353 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9413fe8e-3dfb-4502-be04-d26b3cae42ba-image-registry-private-configuration\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.240713 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.240685 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-registry-tls\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.240914 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.240894 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9413fe8e-3dfb-4502-be04-d26b3cae42ba-installation-pull-secrets\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.250721 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.250695 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzczd\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-kube-api-access-zzczd\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.250846 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.250726 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9413fe8e-3dfb-4502-be04-d26b3cae42ba-bound-sa-token\") pod \"image-registry-646bbcd79b-9792s\" (UID: \"9413fe8e-3dfb-4502-be04-d26b3cae42ba\") " pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:13.263760 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.263732 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" Jul 03 15:14:13.273169 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.273143 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:13.355168 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:13.355138 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:14.436114 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.436083 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646bbcd79b-9792s"] Jul 03 15:14:14.440217 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:14.440189 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9413fe8e_3dfb_4502_be04_d26b3cae42ba.slice/crio-32befb142bf56cd0fe2568d611256eddb8ce13c788316c7ff57dfa6483180e76 WatchSource:0}: Error finding container 32befb142bf56cd0fe2568d611256eddb8ce13c788316c7ff57dfa6483180e76: Status 404 returned error can't find the container with id 32befb142bf56cd0fe2568d611256eddb8ce13c788316c7ff57dfa6483180e76 Jul 03 15:14:14.452231 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.452205 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt"] Jul 03 15:14:14.455509 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:14.455485 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333eb90f_cbae_4ade_9018_9102fed3f9dd.slice/crio-9321020880a5b1e1fee2b7d115e15fccc6cb399f719288c9f248aee8274e12d5 WatchSource:0}: Error finding container 9321020880a5b1e1fee2b7d115e15fccc6cb399f719288c9f248aee8274e12d5: Status 404 returned error can't find the container with id 9321020880a5b1e1fee2b7d115e15fccc6cb399f719288c9f248aee8274e12d5 Jul 03 15:14:14.469852 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.469829 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5"] Jul 03 15:14:14.473270 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:14.473246 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c2172e_7b87_492d_bc80_5f6146a17b9f.slice/crio-39cd6afff64df6af40af0a0a75b4a4af73eec156684b7c4a4791a0ce29ed6e1e WatchSource:0}: Error finding container 39cd6afff64df6af40af0a0a75b4a4af73eec156684b7c4a4791a0ce29ed6e1e: Status 404 returned error can't find the container with id 39cd6afff64df6af40af0a0a75b4a4af73eec156684b7c4a4791a0ce29ed6e1e Jul 03 15:14:14.629585 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.629546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kpqg8" event={"ID":"8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8","Type":"ContainerStarted","Data":"7d82af11168603b4fde615f12c37e9fbff085557c05ca1a034520deec141ecbc"} Jul 03 15:14:14.631139 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.631110 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646bbcd79b-9792s" event={"ID":"9413fe8e-3dfb-4502-be04-d26b3cae42ba","Type":"ContainerStarted","Data":"d5372a65ec48864f1e18c5b3291eb8c07f8dcd18eab3a575d704526833708551"} Jul 03 15:14:14.631259 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.631154 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646bbcd79b-9792s" event={"ID":"9413fe8e-3dfb-4502-be04-d26b3cae42ba","Type":"ContainerStarted","Data":"32befb142bf56cd0fe2568d611256eddb8ce13c788316c7ff57dfa6483180e76"} Jul 03 15:14:14.631259 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.631176 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:14.632234 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.632209 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" event={"ID":"333eb90f-cbae-4ade-9018-9102fed3f9dd","Type":"ContainerStarted","Data":"9321020880a5b1e1fee2b7d115e15fccc6cb399f719288c9f248aee8274e12d5"} Jul 03 15:14:14.633487 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.633465 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" event={"ID":"15c2172e-7b87-492d-bc80-5f6146a17b9f","Type":"ContainerStarted","Data":"39cd6afff64df6af40af0a0a75b4a4af73eec156684b7c4a4791a0ce29ed6e1e"} Jul 03 15:14:14.647344 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.647284 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kpqg8" podStartSLOduration=16.671818778 podStartE2EDuration="24.647260562s" totalImagesPullingTime="7.975441784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:14.647080009 +0000 UTC m=+58.879512900" watchObservedRunningTime="2026-07-03 15:14:14.647260562 +0000 UTC m=+58.879693455" Jul 03 15:14:14.664521 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:14.664474 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-646bbcd79b-9792s" podStartSLOduration=1.6644549629999998 podStartE2EDuration="1.664454963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:14.664224004 +0000 UTC m=+58.896656893" watchObservedRunningTime="2026-07-03 15:14:14.664454963 +0000 UTC m=+58.896887854" Jul 03 15:14:15.532019 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:15.531074 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bzpv8" Jul 03 15:14:20.656916 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.656876 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" event={"ID":"15c2172e-7b87-492d-bc80-5f6146a17b9f","Type":"ContainerStarted","Data":"7608d3e18fc5934542eeb99bc225f4e932404fe52582fadd1133e8fce3076b80"} Jul 03 15:14:20.657437 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.656927 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" event={"ID":"15c2172e-7b87-492d-bc80-5f6146a17b9f","Type":"ContainerStarted","Data":"3b2647c4998001b65019f2e024cb2a8b5e3f0f3fbb04d282671b828c57f955ca"} Jul 03 15:14:20.657437 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.656945 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" event={"ID":"15c2172e-7b87-492d-bc80-5f6146a17b9f","Type":"ContainerStarted","Data":"cf58d4622f797168fd0650a8d010d01d635612f722674515fb553b148b37c6d6"} Jul 03 15:14:20.658367 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.658342 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" event={"ID":"333eb90f-cbae-4ade-9018-9102fed3f9dd","Type":"ContainerStarted","Data":"6d08bb68fcdbe4df39e92651547b7d145f77304e1cd28907db5bcc99d0d58dda"} Jul 03 15:14:20.658548 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.658538 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:20.660140 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.660124 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" Jul 03 15:14:20.685061 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.685007 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b8f977655-lrdd5" podStartSLOduration=3.491456585 podStartE2EDuration="8.684975243s" totalImagesPullingTime="5.193518658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:20.684245927 +0000 UTC m=+64.916678818" watchObservedRunningTime="2026-07-03 15:14:20.684975243 +0000 UTC m=+64.917408144" Jul 03 15:14:20.707090 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:20.707037 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64b8b6868d-v2fnt" podStartSLOduration=3.492387858 podStartE2EDuration="8.707021343s" totalImagesPullingTime="5.214633485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:20.70590399 +0000 UTC m=+64.938336878" watchObservedRunningTime="2026-07-03 15:14:20.707021343 +0000 UTC m=+64.939454232" Jul 03 15:14:21.011554 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.011460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:14:21.011705 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.011554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:14:21.014241 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.014224 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 15:14:21.014307 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.014223 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 15:14:21.024626 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.024590 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec-original-pull-secret\") pod \"global-pull-secret-syncer-c29qj\" (UID: \"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec\") " pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:14:21.024741 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.024637 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb9e801-5990-47ae-964f-6f8e509cffbf-metrics-certs\") pod \"network-metrics-daemon-rjszq\" (UID: \"6eb9e801-5990-47ae-964f-6f8e509cffbf\") " pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:14:21.112445 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.112408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:14:21.115468 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.115445 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 03 15:14:21.125532 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.125506 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 03 15:14:21.135499 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.135479 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bq8p\" (UniqueName: \"kubernetes.io/projected/abe62e1f-4043-470b-827e-6382ee35b8b7-kube-api-access-2bq8p\") pod \"network-check-target-5pbkd\" (UID: \"abe62e1f-4043-470b-827e-6382ee35b8b7\") " pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:14:21.135602 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.135589 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c29qj" Jul 03 15:14:21.145363 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.145343 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9nf84\"" Jul 03 15:14:21.151165 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.151143 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cq8c8\"" Jul 03 15:14:21.153223 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.153205 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:14:21.158626 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.158597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rjszq" Jul 03 15:14:21.285197 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.285166 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c29qj"] Jul 03 15:14:21.288152 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:21.288125 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb9d8dd_f7d0_4f5a_be41_ce3a3ab13bec.slice/crio-efbd95ba271cfed7378784afe5563ad6a884514c0b2e4aaab660cc8d7527bdee WatchSource:0}: Error finding container efbd95ba271cfed7378784afe5563ad6a884514c0b2e4aaab660cc8d7527bdee: Status 404 returned error can't find the container with id efbd95ba271cfed7378784afe5563ad6a884514c0b2e4aaab660cc8d7527bdee Jul 03 15:14:21.301850 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.301820 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5pbkd"] Jul 03 15:14:21.304580 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:21.304555 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe62e1f_4043_470b_827e_6382ee35b8b7.slice/crio-4cbb5bfdfc74fb689405b5bdb2d097ae71d10a20d4ea79c0d2cefce67193fe3c WatchSource:0}: Error finding container 4cbb5bfdfc74fb689405b5bdb2d097ae71d10a20d4ea79c0d2cefce67193fe3c: Status 404 returned error can't find the container with id 4cbb5bfdfc74fb689405b5bdb2d097ae71d10a20d4ea79c0d2cefce67193fe3c Jul 03 15:14:21.320400 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.320377 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rjszq"] Jul 03 15:14:21.323232 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:21.323212 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb9e801_5990_47ae_964f_6f8e509cffbf.slice/crio-1ef9063b66a825556b5bb0b57add06e85c95a0065e86fe10e11276347839d726 WatchSource:0}: Error finding container 1ef9063b66a825556b5bb0b57add06e85c95a0065e86fe10e11276347839d726: Status 404 returned error can't find the container with id 1ef9063b66a825556b5bb0b57add06e85c95a0065e86fe10e11276347839d726 Jul 03 15:14:21.622593 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.622567 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2wrgx" Jul 03 15:14:21.663980 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.663941 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5pbkd" event={"ID":"abe62e1f-4043-470b-827e-6382ee35b8b7","Type":"ContainerStarted","Data":"4cbb5bfdfc74fb689405b5bdb2d097ae71d10a20d4ea79c0d2cefce67193fe3c"} Jul 03 15:14:21.665783 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.665748 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rjszq" event={"ID":"6eb9e801-5990-47ae-964f-6f8e509cffbf","Type":"ContainerStarted","Data":"1ef9063b66a825556b5bb0b57add06e85c95a0065e86fe10e11276347839d726"} Jul 03 15:14:21.667255 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:21.667201 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c29qj" event={"ID":"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec","Type":"ContainerStarted","Data":"efbd95ba271cfed7378784afe5563ad6a884514c0b2e4aaab660cc8d7527bdee"} Jul 03 15:14:22.672848 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:22.672747 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rjszq" event={"ID":"6eb9e801-5990-47ae-964f-6f8e509cffbf","Type":"ContainerStarted","Data":"42964d9ab4ddfa8bbb53ac792feba07b827c9f2f593f09b740adb60fe3f0753c"} Jul 03 15:14:22.672848 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:22.672818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rjszq" event={"ID":"6eb9e801-5990-47ae-964f-6f8e509cffbf","Type":"ContainerStarted","Data":"023480c8f07cd6aa1829ec35780a695f6923b8f110e553f4d8a744abd3845bd6"} Jul 03 15:14:22.689372 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:22.689309 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rjszq" podStartSLOduration=65.738678746 podStartE2EDuration="1m6.689287014s" totalImagesPullingTime="950.608268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:22.688307947 +0000 UTC m=+66.920740837" watchObservedRunningTime="2026-07-03 15:14:22.689287014 +0000 UTC m=+66.921719905" Jul 03 15:14:22.935398 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:22.935312 2572 patch_prober.go:28] interesting pod/image-registry-546bfb8bc9-2mttx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 03 15:14:22.935668 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:22.935633 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" podUID="c2ba0c68-6bca-41e6-9915-077e0dd35a08" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 03 15:14:26.688187 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:26.688153 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5pbkd" event={"ID":"abe62e1f-4043-470b-827e-6382ee35b8b7","Type":"ContainerStarted","Data":"fe9b3ed8bb7e88db4c8420404798f732538b0984cfcf1bb919729608b9376a26"} Jul 03 15:14:26.688532 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:26.688298 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:14:26.704057 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:26.704009 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5pbkd" podStartSLOduration=65.464069867 podStartE2EDuration="1m10.703970573s" totalImagesPullingTime="5.239900706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:26.702691424 +0000 UTC m=+70.935124313" watchObservedRunningTime="2026-07-03 15:14:26.703970573 +0000 UTC m=+70.936403460" Jul 03 15:14:27.205211 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.205177 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t96ln"] Jul 03 15:14:27.207419 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.207402 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.212662 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.212630 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 03 15:14:27.212662 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.212663 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 03 15:14:27.213142 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.213123 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2mmx\"" Jul 03 15:14:27.213242 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.213123 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 03 15:14:27.213242 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.213215 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 03 15:14:27.213336 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.213123 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 03 15:14:27.213388 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.213364 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 03 15:14:27.264055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264015 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-tls\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264258 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264079 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhz9d\" (UniqueName: \"kubernetes.io/projected/702b7539-b6d5-405a-8142-93b83e360345-kube-api-access-qhz9d\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264258 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-sys\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264372 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-textfile\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264372 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-wtmp\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264372 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264331 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264517 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-root\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264517 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/702b7539-b6d5-405a-8142-93b83e360345-metrics-client-ca\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.264517 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.264468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-accelerators-collector-config\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.364842 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.364798 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-sys\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.364842 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.364835 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-textfile\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.364856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-wtmp\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.364881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.364920 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-sys\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.364937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-root\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.364960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/702b7539-b6d5-405a-8142-93b83e360345-metrics-client-ca\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-accelerators-collector-config\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-tls\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365041 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-wtmp\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365132 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhz9d\" (UniqueName: \"kubernetes.io/projected/702b7539-b6d5-405a-8142-93b83e360345-kube-api-access-qhz9d\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365492 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365316 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-textfile\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365492 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365027 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/702b7539-b6d5-405a-8142-93b83e360345-root\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365693 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365667 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-accelerators-collector-config\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.365791 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.365773 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/702b7539-b6d5-405a-8142-93b83e360345-metrics-client-ca\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.367282 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.367261 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.367444 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.367426 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/702b7539-b6d5-405a-8142-93b83e360345-node-exporter-tls\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.374276 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.374251 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhz9d\" (UniqueName: \"kubernetes.io/projected/702b7539-b6d5-405a-8142-93b83e360345-kube-api-access-qhz9d\") pod \"node-exporter-t96ln\" (UID: \"702b7539-b6d5-405a-8142-93b83e360345\") " pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.519093 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.519012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t96ln" Jul 03 15:14:27.528892 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:14:27.528863 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702b7539_b6d5_405a_8142_93b83e360345.slice/crio-c197864d10a92ccf8556bf600a91bb54686259f04dadd585db3e46b8923ffb50 WatchSource:0}: Error finding container c197864d10a92ccf8556bf600a91bb54686259f04dadd585db3e46b8923ffb50: Status 404 returned error can't find the container with id c197864d10a92ccf8556bf600a91bb54686259f04dadd585db3e46b8923ffb50 Jul 03 15:14:27.693030 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.692975 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t96ln" event={"ID":"702b7539-b6d5-405a-8142-93b83e360345","Type":"ContainerStarted","Data":"c197864d10a92ccf8556bf600a91bb54686259f04dadd585db3e46b8923ffb50"} Jul 03 15:14:27.694448 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.694417 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c29qj" event={"ID":"8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec","Type":"ContainerStarted","Data":"41b685cdc6c022a88985098d8c3de0e4de068cb1c5ac678ad5a5c71ba7b88929"} Jul 03 15:14:27.708778 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:27.708725 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c29qj" podStartSLOduration=66.441718903 podStartE2EDuration="1m11.708711633s" totalImagesPullingTime="5.26699273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:27.708029231 +0000 UTC m=+71.940462123" watchObservedRunningTime="2026-07-03 15:14:27.708711633 +0000 UTC m=+71.941144561" Jul 03 15:14:28.699413 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:28.699377 2572 generic.go:356] "Generic (PLEG): container finished" podID="702b7539-b6d5-405a-8142-93b83e360345" containerID="7250d6cc821b0fe918185fe8b1522ab90db42fc9a0f97258fb3a7522cb533b25" exitCode=0 Jul 03 15:14:28.699827 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:28.699458 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t96ln" event={"ID":"702b7539-b6d5-405a-8142-93b83e360345","Type":"ContainerDied","Data":"7250d6cc821b0fe918185fe8b1522ab90db42fc9a0f97258fb3a7522cb533b25"} Jul 03 15:14:29.704447 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:29.704402 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t96ln" event={"ID":"702b7539-b6d5-405a-8142-93b83e360345","Type":"ContainerStarted","Data":"f62acd1c8c340bf11d689877a69abc2fcbc2d6dade064ab536050733d37507fc"} Jul 03 15:14:29.704800 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:29.704470 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t96ln" event={"ID":"702b7539-b6d5-405a-8142-93b83e360345","Type":"ContainerStarted","Data":"86700038243dc5d5f5b7bd8685ef045e79b2df407b1b62726d1e0332e1ee43c8"} Jul 03 15:14:29.725039 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:29.724968 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t96ln" podStartSLOduration=1.659891303 podStartE2EDuration="2.724953643s" totalImagesPullingTime="1.000176581s" totalInitContainerRuntime="64.885759ms" isStatefulPod=true podCreationTimestamp="2026-07-03 15:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:14:29.723709396 +0000 UTC m=+73.956142285" watchObservedRunningTime="2026-07-03 15:14:29.724953643 +0000 UTC m=+73.957386532" Jul 03 15:14:32.930904 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:32.930877 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:14:35.643166 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:35.643136 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-646bbcd79b-9792s" Jul 03 15:14:37.956019 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:37.955866 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" podUID="c2ba0c68-6bca-41e6-9915-077e0dd35a08" containerName="registry" containerID="cri-o://b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b" gracePeriod=30 Jul 03 15:14:38.200469 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.200447 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:14:38.340398 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340361 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.340551 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340409 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzd8k\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-kube-api-access-wzd8k\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.340551 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340426 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-trusted-ca\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.340551 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340462 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2ba0c68-6bca-41e6-9915-077e0dd35a08-ca-trust-extracted\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.340551 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340479 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-installation-pull-secrets\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.340551 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340515 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-image-registry-private-configuration\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.340798 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340554 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-bound-sa-token\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.340798 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.340582 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-certificates\") pod \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\" (UID: \"c2ba0c68-6bca-41e6-9915-077e0dd35a08\") " Jul 03 15:14:38.341105 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.341078 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 15:14:38.341344 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.341306 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 15:14:38.343226 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.343189 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:14:38.343333 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.343261 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:14:38.343333 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.343310 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:14:38.343454 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.343366 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-kube-api-access-wzd8k" (OuterVolumeSpecName: "kube-api-access-wzd8k") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "kube-api-access-wzd8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:14:38.343538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.343477 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:14:38.348834 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.348807 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ba0c68-6bca-41e6-9915-077e0dd35a08-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c2ba0c68-6bca-41e6-9915-077e0dd35a08" (UID: "c2ba0c68-6bca-41e6-9915-077e0dd35a08"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:14:38.441853 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441817 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.441853 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441846 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzd8k\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-kube-api-access-wzd8k\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.441853 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441858 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-trusted-ca\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.442104 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441867 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2ba0c68-6bca-41e6-9915-077e0dd35a08-ca-trust-extracted\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.442104 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441875 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-installation-pull-secrets\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.442104 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441885 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2ba0c68-6bca-41e6-9915-077e0dd35a08-image-registry-private-configuration\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.442104 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441896 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ba0c68-6bca-41e6-9915-077e0dd35a08-bound-sa-token\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.442104 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.441903 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2ba0c68-6bca-41e6-9915-077e0dd35a08-registry-certificates\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:14:38.733561 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.733471 2572 generic.go:356] "Generic (PLEG): container finished" podID="c2ba0c68-6bca-41e6-9915-077e0dd35a08" containerID="b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b" exitCode=0 Jul 03 15:14:38.733561 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.733522 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" Jul 03 15:14:38.733561 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.733544 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" event={"ID":"c2ba0c68-6bca-41e6-9915-077e0dd35a08","Type":"ContainerDied","Data":"b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b"} Jul 03 15:14:38.733830 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.733582 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546bfb8bc9-2mttx" event={"ID":"c2ba0c68-6bca-41e6-9915-077e0dd35a08","Type":"ContainerDied","Data":"d565d45a537bd8f8d9b8e926b5bf26aae1762a8fb62e152ee88323eae0df3357"} Jul 03 15:14:38.733830 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.733601 2572 scope.go:117] "RemoveContainer" containerID="b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b" Jul 03 15:14:38.741766 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.741749 2572 scope.go:117] "RemoveContainer" containerID="b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b" Jul 03 15:14:38.742050 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:14:38.742019 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b\": container with ID starting with b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b not found: ID does not exist" containerID="b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b" Jul 03 15:14:38.742131 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.742057 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b"} err="failed to get container status \"b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b\": rpc error: code = NotFound desc = could not find container \"b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b\": container with ID starting with b10d5fbafc5b6503b1ada44021026e2639907ec78697319202cbf3a726d5850b not found: ID does not exist" Jul 03 15:14:38.753702 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.753677 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546bfb8bc9-2mttx"] Jul 03 15:14:38.760444 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:38.760422 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-546bfb8bc9-2mttx"] Jul 03 15:14:40.229276 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:40.229242 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ba0c68-6bca-41e6-9915-077e0dd35a08" path="/var/lib/kubelet/pods/c2ba0c68-6bca-41e6-9915-077e0dd35a08/volumes" Jul 03 15:14:57.697720 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:14:57.697692 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5pbkd" Jul 03 15:17:51.172763 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.172688 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc"] Jul 03 15:17:51.173247 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.172928 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ba0c68-6bca-41e6-9915-077e0dd35a08" containerName="registry" Jul 03 15:17:51.173247 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.172938 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ba0c68-6bca-41e6-9915-077e0dd35a08" containerName="registry" Jul 03 15:17:51.173247 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.172977 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ba0c68-6bca-41e6-9915-077e0dd35a08" containerName="registry" Jul 03 15:17:51.175820 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.175796 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.178846 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.178823 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 03 15:17:51.179374 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.179356 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bf577\"" Jul 03 15:17:51.179934 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.179918 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 03 15:17:51.189156 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.189136 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc"] Jul 03 15:17:51.310769 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.310729 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.310934 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.310792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.310934 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.310822 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jb2\" (UniqueName: \"kubernetes.io/projected/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-kube-api-access-j4jb2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.411393 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.411347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.411580 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.411407 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.411580 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.411453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jb2\" (UniqueName: \"kubernetes.io/projected/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-kube-api-access-j4jb2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.411837 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.411816 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.411883 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.411839 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.419873 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.419841 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jb2\" (UniqueName: \"kubernetes.io/projected/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-kube-api-access-j4jb2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.484424 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.484339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:17:51.610561 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:51.610525 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc"] Jul 03 15:17:51.614242 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:17:51.614214 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97a20bb_f1c9_41cb_9e34_edc8fd365d79.slice/crio-b441d6060fd815d260d06fb3897de4c425f6f5347d11f8aa22bd9ee511b9be96 WatchSource:0}: Error finding container b441d6060fd815d260d06fb3897de4c425f6f5347d11f8aa22bd9ee511b9be96: Status 404 returned error can't find the container with id b441d6060fd815d260d06fb3897de4c425f6f5347d11f8aa22bd9ee511b9be96 Jul 03 15:17:52.349470 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:52.349427 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" event={"ID":"b97a20bb-f1c9-41cb-9e34-edc8fd365d79","Type":"ContainerStarted","Data":"b441d6060fd815d260d06fb3897de4c425f6f5347d11f8aa22bd9ee511b9be96"} Jul 03 15:17:58.373889 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:58.373842 2572 generic.go:356] "Generic (PLEG): container finished" podID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerID="951599e363eb412a1319b7498e0605e7ad5772a994bd291e997a1500a42065a5" exitCode=0 Jul 03 15:17:58.374278 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:17:58.373921 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" event={"ID":"b97a20bb-f1c9-41cb-9e34-edc8fd365d79","Type":"ContainerDied","Data":"951599e363eb412a1319b7498e0605e7ad5772a994bd291e997a1500a42065a5"} Jul 03 15:18:09.411069 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:09.411034 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" event={"ID":"b97a20bb-f1c9-41cb-9e34-edc8fd365d79","Type":"ContainerStarted","Data":"c9b0105ad263a920ea1e4737774598fd974cb84289b76278e0f7538e68846790"} Jul 03 15:18:10.415707 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:10.415674 2572 generic.go:356] "Generic (PLEG): container finished" podID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerID="c9b0105ad263a920ea1e4737774598fd974cb84289b76278e0f7538e68846790" exitCode=0 Jul 03 15:18:10.416207 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:10.415743 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" event={"ID":"b97a20bb-f1c9-41cb-9e34-edc8fd365d79","Type":"ContainerDied","Data":"c9b0105ad263a920ea1e4737774598fd974cb84289b76278e0f7538e68846790"} Jul 03 15:18:17.380944 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:17.380917 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:18:17.382537 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:17.381167 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:18:17.383429 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:17.383414 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 03 15:18:18.448550 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:18.448511 2572 generic.go:356] "Generic (PLEG): container finished" podID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerID="810074109bca5f10d701ac1b8ca7ce483e9530515a594448ac40a8dc3bb3b6ce" exitCode=0 Jul 03 15:18:18.448934 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:18.448596 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" event={"ID":"b97a20bb-f1c9-41cb-9e34-edc8fd365d79","Type":"ContainerDied","Data":"810074109bca5f10d701ac1b8ca7ce483e9530515a594448ac40a8dc3bb3b6ce"} Jul 03 15:18:19.572980 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.572955 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:18:19.734278 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.734197 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-bundle\") pod \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " Jul 03 15:18:19.734278 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.734251 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-util\") pod \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " Jul 03 15:18:19.734452 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.734310 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4jb2\" (UniqueName: \"kubernetes.io/projected/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-kube-api-access-j4jb2\") pod \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\" (UID: \"b97a20bb-f1c9-41cb-9e34-edc8fd365d79\") " Jul 03 15:18:19.734882 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.734858 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-bundle" (OuterVolumeSpecName: "bundle") pod "b97a20bb-f1c9-41cb-9e34-edc8fd365d79" (UID: "b97a20bb-f1c9-41cb-9e34-edc8fd365d79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:18:19.736475 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.736444 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-kube-api-access-j4jb2" (OuterVolumeSpecName: "kube-api-access-j4jb2") pod "b97a20bb-f1c9-41cb-9e34-edc8fd365d79" (UID: "b97a20bb-f1c9-41cb-9e34-edc8fd365d79"). InnerVolumeSpecName "kube-api-access-j4jb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:18:19.739010 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.738975 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-util" (OuterVolumeSpecName: "util") pod "b97a20bb-f1c9-41cb-9e34-edc8fd365d79" (UID: "b97a20bb-f1c9-41cb-9e34-edc8fd365d79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:18:19.835092 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.835059 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jb2\" (UniqueName: \"kubernetes.io/projected/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-kube-api-access-j4jb2\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:19.835092 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.835086 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:19.835092 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:19.835096 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b97a20bb-f1c9-41cb-9e34-edc8fd365d79-util\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:20.456355 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:20.456319 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" event={"ID":"b97a20bb-f1c9-41cb-9e34-edc8fd365d79","Type":"ContainerDied","Data":"b441d6060fd815d260d06fb3897de4c425f6f5347d11f8aa22bd9ee511b9be96"} Jul 03 15:18:20.456514 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:20.456376 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b441d6060fd815d260d06fb3897de4c425f6f5347d11f8aa22bd9ee511b9be96" Jul 03 15:18:20.456514 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:20.456333 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fkkcc" Jul 03 15:18:28.410357 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410318 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb"] Jul 03 15:18:28.410844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410574 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerName="extract" Jul 03 15:18:28.410844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410586 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerName="extract" Jul 03 15:18:28.410844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410594 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerName="pull" Jul 03 15:18:28.410844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410600 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerName="pull" Jul 03 15:18:28.410844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410609 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerName="util" Jul 03 15:18:28.410844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410614 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerName="util" Jul 03 15:18:28.410844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.410654 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97a20bb-f1c9-41cb-9e34-edc8fd365d79" containerName="extract" Jul 03 15:18:28.413777 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.413761 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.416537 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.416513 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 03 15:18:28.417197 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.417177 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bf577\"" Jul 03 15:18:28.417618 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.417600 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 03 15:18:28.421701 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.421672 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb"] Jul 03 15:18:28.598778 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.598744 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.598778 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.598777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.599011 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.598804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sq65\" (UniqueName: \"kubernetes.io/projected/dec777c5-ae6c-4369-8ca8-d3ac91142e00-kube-api-access-4sq65\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.700202 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.700116 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.700202 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.700153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.700202 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.700191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sq65\" (UniqueName: \"kubernetes.io/projected/dec777c5-ae6c-4369-8ca8-d3ac91142e00-kube-api-access-4sq65\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.700514 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.700491 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.700579 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.700558 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.716574 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.716548 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sq65\" (UniqueName: \"kubernetes.io/projected/dec777c5-ae6c-4369-8ca8-d3ac91142e00-kube-api-access-4sq65\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.723311 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.723293 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:28.868866 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:28.868822 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb"] Jul 03 15:18:28.871615 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:18:28.871587 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec777c5_ae6c_4369_8ca8_d3ac91142e00.slice/crio-9385eb3a21a04e6d43b0c72196877a74e1360f3b3fcc53f885707c839516128d WatchSource:0}: Error finding container 9385eb3a21a04e6d43b0c72196877a74e1360f3b3fcc53f885707c839516128d: Status 404 returned error can't find the container with id 9385eb3a21a04e6d43b0c72196877a74e1360f3b3fcc53f885707c839516128d Jul 03 15:18:29.488241 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:29.488204 2572 generic.go:356] "Generic (PLEG): container finished" podID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerID="a74008e99def8e7f4ac03f55c6c683acf2aed898561ef03e3448e7544d130d1b" exitCode=0 Jul 03 15:18:29.488241 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:29.488247 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" event={"ID":"dec777c5-ae6c-4369-8ca8-d3ac91142e00","Type":"ContainerDied","Data":"a74008e99def8e7f4ac03f55c6c683acf2aed898561ef03e3448e7544d130d1b"} Jul 03 15:18:29.488653 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:29.488292 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" event={"ID":"dec777c5-ae6c-4369-8ca8-d3ac91142e00","Type":"ContainerStarted","Data":"9385eb3a21a04e6d43b0c72196877a74e1360f3b3fcc53f885707c839516128d"} Jul 03 15:18:29.489283 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:29.489267 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 15:18:32.500961 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:32.500928 2572 generic.go:356] "Generic (PLEG): container finished" podID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerID="a7c01866c44f6dc54afa3cb91c9f55d28bc942e306b0f8a759f3ee6c1389e6fc" exitCode=0 Jul 03 15:18:32.501371 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:32.501007 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" event={"ID":"dec777c5-ae6c-4369-8ca8-d3ac91142e00","Type":"ContainerDied","Data":"a7c01866c44f6dc54afa3cb91c9f55d28bc942e306b0f8a759f3ee6c1389e6fc"} Jul 03 15:18:33.506073 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:33.506037 2572 generic.go:356] "Generic (PLEG): container finished" podID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerID="e06a0f04f6456b94c6ce7409eb83381a488eb59ad510ee69d8cfe5bfb57b329f" exitCode=0 Jul 03 15:18:33.506442 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:33.506129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" event={"ID":"dec777c5-ae6c-4369-8ca8-d3ac91142e00","Type":"ContainerDied","Data":"e06a0f04f6456b94c6ce7409eb83381a488eb59ad510ee69d8cfe5bfb57b329f"} Jul 03 15:18:34.638257 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.638235 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:34.640483 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.640465 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-bundle\") pod \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " Jul 03 15:18:34.640572 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.640494 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-util\") pod \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " Jul 03 15:18:34.640572 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.640530 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sq65\" (UniqueName: \"kubernetes.io/projected/dec777c5-ae6c-4369-8ca8-d3ac91142e00-kube-api-access-4sq65\") pod \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\" (UID: \"dec777c5-ae6c-4369-8ca8-d3ac91142e00\") " Jul 03 15:18:34.640881 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.640858 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-bundle" (OuterVolumeSpecName: "bundle") pod "dec777c5-ae6c-4369-8ca8-d3ac91142e00" (UID: "dec777c5-ae6c-4369-8ca8-d3ac91142e00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:18:34.642635 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.642610 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec777c5-ae6c-4369-8ca8-d3ac91142e00-kube-api-access-4sq65" (OuterVolumeSpecName: "kube-api-access-4sq65") pod "dec777c5-ae6c-4369-8ca8-d3ac91142e00" (UID: "dec777c5-ae6c-4369-8ca8-d3ac91142e00"). InnerVolumeSpecName "kube-api-access-4sq65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:18:34.645307 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.645265 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-util" (OuterVolumeSpecName: "util") pod "dec777c5-ae6c-4369-8ca8-d3ac91142e00" (UID: "dec777c5-ae6c-4369-8ca8-d3ac91142e00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:18:34.741656 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.741615 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:34.741656 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.741648 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dec777c5-ae6c-4369-8ca8-d3ac91142e00-util\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:34.741656 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:34.741657 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sq65\" (UniqueName: \"kubernetes.io/projected/dec777c5-ae6c-4369-8ca8-d3ac91142e00-kube-api-access-4sq65\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:35.513978 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:35.513937 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" event={"ID":"dec777c5-ae6c-4369-8ca8-d3ac91142e00","Type":"ContainerDied","Data":"9385eb3a21a04e6d43b0c72196877a74e1360f3b3fcc53f885707c839516128d"} Jul 03 15:18:35.513978 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:35.513951 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f87dnb" Jul 03 15:18:35.514200 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:35.514053 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9385eb3a21a04e6d43b0c72196877a74e1360f3b3fcc53f885707c839516128d" Jul 03 15:18:40.547024 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.546977 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47"] Jul 03 15:18:40.547432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.547215 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerName="extract" Jul 03 15:18:40.547432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.547225 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerName="extract" Jul 03 15:18:40.547432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.547238 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerName="pull" Jul 03 15:18:40.547432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.547243 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerName="pull" Jul 03 15:18:40.547432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.547258 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerName="util" Jul 03 15:18:40.547432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.547263 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerName="util" Jul 03 15:18:40.547432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.547303 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec777c5-ae6c-4369-8ca8-d3ac91142e00" containerName="extract" Jul 03 15:18:40.549940 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.549923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:40.552530 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.552510 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 03 15:18:40.552656 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.552558 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 03 15:18:40.553756 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.553740 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-sj86q\"" Jul 03 15:18:40.557484 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.557446 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47"] Jul 03 15:18:40.581889 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.581856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3d54199-d2d9-4640-aa3b-8a7d24c70c9c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z7m47\" (UID: \"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:40.582040 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.581912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9sq\" (UniqueName: \"kubernetes.io/projected/e3d54199-d2d9-4640-aa3b-8a7d24c70c9c-kube-api-access-vx9sq\") pod \"openshift-lws-operator-fd8ccff4c-z7m47\" (UID: \"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:40.682729 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.682697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3d54199-d2d9-4640-aa3b-8a7d24c70c9c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z7m47\" (UID: \"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:40.682875 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.682768 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9sq\" (UniqueName: \"kubernetes.io/projected/e3d54199-d2d9-4640-aa3b-8a7d24c70c9c-kube-api-access-vx9sq\") pod \"openshift-lws-operator-fd8ccff4c-z7m47\" (UID: \"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:40.683168 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.683150 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3d54199-d2d9-4640-aa3b-8a7d24c70c9c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z7m47\" (UID: \"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:40.691638 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.691611 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9sq\" (UniqueName: \"kubernetes.io/projected/e3d54199-d2d9-4640-aa3b-8a7d24c70c9c-kube-api-access-vx9sq\") pod \"openshift-lws-operator-fd8ccff4c-z7m47\" (UID: \"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:40.859735 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:40.859703 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" Jul 03 15:18:41.007084 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:41.007048 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47"] Jul 03 15:18:41.010507 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:18:41.010479 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3d54199_d2d9_4640_aa3b_8a7d24c70c9c.slice/crio-5759ea5174cc022d850170026ffd27664ba1cb87e3b670120fac558c5a274659 WatchSource:0}: Error finding container 5759ea5174cc022d850170026ffd27664ba1cb87e3b670120fac558c5a274659: Status 404 returned error can't find the container with id 5759ea5174cc022d850170026ffd27664ba1cb87e3b670120fac558c5a274659 Jul 03 15:18:41.535278 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:41.535240 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" event={"ID":"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c","Type":"ContainerStarted","Data":"5759ea5174cc022d850170026ffd27664ba1cb87e3b670120fac558c5a274659"} Jul 03 15:18:43.544512 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:43.544481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" event={"ID":"e3d54199-d2d9-4640-aa3b-8a7d24c70c9c","Type":"ContainerStarted","Data":"7a20063666d246a7f6d5a1d1c39be1237ae6b7588a0484a340085f626e19296d"} Jul 03 15:18:43.559956 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:43.559872 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7m47" podStartSLOduration=1.821069147 podStartE2EDuration="3.559851036s" totalImagesPullingTime="1.738781889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:18:43.559071874 +0000 UTC m=+327.791504763" watchObservedRunningTime="2026-07-03 15:18:43.559851036 +0000 UTC m=+327.792283929" Jul 03 15:18:46.590704 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.590669 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr"] Jul 03 15:18:46.594023 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.594006 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.598313 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.598283 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 03 15:18:46.598444 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.598429 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bf577\"" Jul 03 15:18:46.599620 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.599599 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 03 15:18:46.618851 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.618823 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqz4m\" (UniqueName: \"kubernetes.io/projected/de9a91ec-c870-441e-9c5b-762df168262c-kube-api-access-qqz4m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.619024 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.618872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.619024 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.618900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.620647 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.620622 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr"] Jul 03 15:18:46.719281 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.719235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqz4m\" (UniqueName: \"kubernetes.io/projected/de9a91ec-c870-441e-9c5b-762df168262c-kube-api-access-qqz4m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.719488 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.719293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.719488 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.719318 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.719677 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.719654 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.719736 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.719693 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.729801 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.729776 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqz4m\" (UniqueName: \"kubernetes.io/projected/de9a91ec-c870-441e-9c5b-762df168262c-kube-api-access-qqz4m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:46.903316 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:46.903230 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:47.051258 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:47.051220 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr"] Jul 03 15:18:47.053923 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:18:47.053897 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9a91ec_c870_441e_9c5b_762df168262c.slice/crio-d63391a26f9ec3634927e84fce296366c32f3116e13057b8b5fc28ff060e3aa4 WatchSource:0}: Error finding container d63391a26f9ec3634927e84fce296366c32f3116e13057b8b5fc28ff060e3aa4: Status 404 returned error can't find the container with id d63391a26f9ec3634927e84fce296366c32f3116e13057b8b5fc28ff060e3aa4 Jul 03 15:18:47.560292 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:47.560200 2572 generic.go:356] "Generic (PLEG): container finished" podID="de9a91ec-c870-441e-9c5b-762df168262c" containerID="24f9787cec79f2bc71d024aa4c85580d3c3cc4f36184be0667bdd9268f09633d" exitCode=0 Jul 03 15:18:47.560292 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:47.560262 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" event={"ID":"de9a91ec-c870-441e-9c5b-762df168262c","Type":"ContainerDied","Data":"24f9787cec79f2bc71d024aa4c85580d3c3cc4f36184be0667bdd9268f09633d"} Jul 03 15:18:47.560524 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:47.560314 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" event={"ID":"de9a91ec-c870-441e-9c5b-762df168262c","Type":"ContainerStarted","Data":"d63391a26f9ec3634927e84fce296366c32f3116e13057b8b5fc28ff060e3aa4"} Jul 03 15:18:49.569060 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:49.569026 2572 generic.go:356] "Generic (PLEG): container finished" podID="de9a91ec-c870-441e-9c5b-762df168262c" containerID="a665dc25c782c01411cdc345889c0aac084803cf2045906c426fff0cef73326f" exitCode=0 Jul 03 15:18:49.569532 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:49.569114 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" event={"ID":"de9a91ec-c870-441e-9c5b-762df168262c","Type":"ContainerDied","Data":"a665dc25c782c01411cdc345889c0aac084803cf2045906c426fff0cef73326f"} Jul 03 15:18:50.574982 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:50.574947 2572 generic.go:356] "Generic (PLEG): container finished" podID="de9a91ec-c870-441e-9c5b-762df168262c" containerID="5eb487c3a19572e19afa3b438e5de2ad134975f248ea839147891d17cc2419c2" exitCode=0 Jul 03 15:18:50.575376 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:50.575031 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" event={"ID":"de9a91ec-c870-441e-9c5b-762df168262c","Type":"ContainerDied","Data":"5eb487c3a19572e19afa3b438e5de2ad134975f248ea839147891d17cc2419c2"} Jul 03 15:18:51.697727 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.697701 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:51.755921 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.755884 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-bundle\") pod \"de9a91ec-c870-441e-9c5b-762df168262c\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " Jul 03 15:18:51.755921 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.755918 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-util\") pod \"de9a91ec-c870-441e-9c5b-762df168262c\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " Jul 03 15:18:51.756138 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.755951 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqz4m\" (UniqueName: \"kubernetes.io/projected/de9a91ec-c870-441e-9c5b-762df168262c-kube-api-access-qqz4m\") pod \"de9a91ec-c870-441e-9c5b-762df168262c\" (UID: \"de9a91ec-c870-441e-9c5b-762df168262c\") " Jul 03 15:18:51.756670 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.756642 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-bundle" (OuterVolumeSpecName: "bundle") pod "de9a91ec-c870-441e-9c5b-762df168262c" (UID: "de9a91ec-c870-441e-9c5b-762df168262c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:18:51.757979 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.757950 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9a91ec-c870-441e-9c5b-762df168262c-kube-api-access-qqz4m" (OuterVolumeSpecName: "kube-api-access-qqz4m") pod "de9a91ec-c870-441e-9c5b-762df168262c" (UID: "de9a91ec-c870-441e-9c5b-762df168262c"). InnerVolumeSpecName "kube-api-access-qqz4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:18:51.761472 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.761432 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-util" (OuterVolumeSpecName: "util") pod "de9a91ec-c870-441e-9c5b-762df168262c" (UID: "de9a91ec-c870-441e-9c5b-762df168262c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:18:51.856821 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.856781 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqz4m\" (UniqueName: \"kubernetes.io/projected/de9a91ec-c870-441e-9c5b-762df168262c-kube-api-access-qqz4m\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:51.856821 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.856823 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:51.856821 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:51.856833 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9a91ec-c870-441e-9c5b-762df168262c-util\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:18:52.583574 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:52.583551 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" Jul 03 15:18:52.583574 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:52.583566 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlpwr" event={"ID":"de9a91ec-c870-441e-9c5b-762df168262c","Type":"ContainerDied","Data":"d63391a26f9ec3634927e84fce296366c32f3116e13057b8b5fc28ff060e3aa4"} Jul 03 15:18:52.583805 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:18:52.583607 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d63391a26f9ec3634927e84fce296366c32f3116e13057b8b5fc28ff060e3aa4" Jul 03 15:19:04.296965 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.296934 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5"] Jul 03 15:19:04.297392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.297179 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de9a91ec-c870-441e-9c5b-762df168262c" containerName="util" Jul 03 15:19:04.297392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.297190 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de9a91ec-c870-441e-9c5b-762df168262c" containerName="util" Jul 03 15:19:04.297392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.297200 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de9a91ec-c870-441e-9c5b-762df168262c" containerName="extract" Jul 03 15:19:04.297392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.297205 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de9a91ec-c870-441e-9c5b-762df168262c" containerName="extract" Jul 03 15:19:04.297392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.297221 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de9a91ec-c870-441e-9c5b-762df168262c" containerName="pull" Jul 03 15:19:04.297392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.297227 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de9a91ec-c870-441e-9c5b-762df168262c" containerName="pull" Jul 03 15:19:04.297392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.297265 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de9a91ec-c870-441e-9c5b-762df168262c" containerName="extract" Jul 03 15:19:04.300310 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.300292 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.303639 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.303591 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 03 15:19:04.303775 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.303706 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 03 15:19:04.303962 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.303948 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 03 15:19:04.304254 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.304237 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4hhbk\"" Jul 03 15:19:04.304335 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.304242 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 03 15:19:04.325668 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.325637 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5"] Jul 03 15:19:04.343667 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.343629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36340a31-6ebd-40b8-84f3-430c400a0091-webhook-cert\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.343826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.343673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36340a31-6ebd-40b8-84f3-430c400a0091-tmp\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.343826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.343703 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36340a31-6ebd-40b8-84f3-430c400a0091-manifests\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.343826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.343796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jvf\" (UniqueName: \"kubernetes.io/projected/36340a31-6ebd-40b8-84f3-430c400a0091-kube-api-access-26jvf\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.343931 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.343830 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36340a31-6ebd-40b8-84f3-430c400a0091-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.444887 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.444854 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jvf\" (UniqueName: \"kubernetes.io/projected/36340a31-6ebd-40b8-84f3-430c400a0091-kube-api-access-26jvf\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.444887 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.444893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36340a31-6ebd-40b8-84f3-430c400a0091-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.445182 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.444919 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36340a31-6ebd-40b8-84f3-430c400a0091-webhook-cert\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.445182 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.444938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36340a31-6ebd-40b8-84f3-430c400a0091-tmp\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.445182 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.444957 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36340a31-6ebd-40b8-84f3-430c400a0091-manifests\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.445474 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.445455 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36340a31-6ebd-40b8-84f3-430c400a0091-manifests\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.447360 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.447333 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36340a31-6ebd-40b8-84f3-430c400a0091-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.447699 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.447680 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36340a31-6ebd-40b8-84f3-430c400a0091-tmp\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.447782 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.447765 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36340a31-6ebd-40b8-84f3-430c400a0091-webhook-cert\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.455909 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.455885 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jvf\" (UniqueName: \"kubernetes.io/projected/36340a31-6ebd-40b8-84f3-430c400a0091-kube-api-access-26jvf\") pod \"opendatahub-operator-controller-manager-86b59fc499-f5wg5\" (UID: \"36340a31-6ebd-40b8-84f3-430c400a0091\") " pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.610245 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.610217 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:04.748082 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:04.748047 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5"] Jul 03 15:19:04.751330 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:19:04.751303 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36340a31_6ebd_40b8_84f3_430c400a0091.slice/crio-901dbb9dd0ef969b8988bfa7d4c334a8b7f80f8d55b662b2e1258a8d20dc83da WatchSource:0}: Error finding container 901dbb9dd0ef969b8988bfa7d4c334a8b7f80f8d55b662b2e1258a8d20dc83da: Status 404 returned error can't find the container with id 901dbb9dd0ef969b8988bfa7d4c334a8b7f80f8d55b662b2e1258a8d20dc83da Jul 03 15:19:05.633016 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:05.632560 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" event={"ID":"36340a31-6ebd-40b8-84f3-430c400a0091","Type":"ContainerStarted","Data":"901dbb9dd0ef969b8988bfa7d4c334a8b7f80f8d55b662b2e1258a8d20dc83da"} Jul 03 15:19:08.648594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:08.648552 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" event={"ID":"36340a31-6ebd-40b8-84f3-430c400a0091","Type":"ContainerStarted","Data":"c10ee7b7604e87dffbbe9e6c4cc6a636eedb65117676673a023d3219f4cb8b12"} Jul 03 15:19:09.657293 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:09.657252 2572 generic.go:356] "Generic (PLEG): container finished" podID="36340a31-6ebd-40b8-84f3-430c400a0091" containerID="c10ee7b7604e87dffbbe9e6c4cc6a636eedb65117676673a023d3219f4cb8b12" exitCode=0 Jul 03 15:19:09.657765 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:09.657322 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" event={"ID":"36340a31-6ebd-40b8-84f3-430c400a0091","Type":"ContainerDied","Data":"c10ee7b7604e87dffbbe9e6c4cc6a636eedb65117676673a023d3219f4cb8b12"} Jul 03 15:19:10.668734 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:10.668696 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" event={"ID":"36340a31-6ebd-40b8-84f3-430c400a0091","Type":"ContainerStarted","Data":"29c20eb5297ed84faf5398b1016dd38b5ccc6966bd798e60964e689b4c7149b6"} Jul 03 15:19:10.669147 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:10.668863 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:10.690245 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:10.690193 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" podStartSLOduration=2.310807571 podStartE2EDuration="6.690178555s" totalImagesPullingTime="3.316871086s" totalInitContainerRuntime="1.062499898s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:19:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:19:10.688413088 +0000 UTC m=+354.920845976" watchObservedRunningTime="2026-07-03 15:19:10.690178555 +0000 UTC m=+354.922611443" Jul 03 15:19:21.675658 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:21.675631 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86b59fc499-f5wg5" Jul 03 15:19:24.207237 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.207196 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9d497994-8fhn2"] Jul 03 15:19:24.215740 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.215713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.219353 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.219328 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 03 15:19:24.219508 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.219381 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 03 15:19:24.219508 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.219420 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 03 15:19:24.219771 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.219755 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 03 15:19:24.230270 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.230247 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9d497994-8fhn2"] Jul 03 15:19:24.304044 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.304012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b42884e-e056-4bc3-85f7-d2879960b3fb-tmp\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.304234 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.304062 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8pl\" (UniqueName: \"kubernetes.io/projected/0b42884e-e056-4bc3-85f7-d2879960b3fb-kube-api-access-bp8pl\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.304234 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.304144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b42884e-e056-4bc3-85f7-d2879960b3fb-tls-certs\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.405482 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.405434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b42884e-e056-4bc3-85f7-d2879960b3fb-tmp\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.405482 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.405489 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8pl\" (UniqueName: \"kubernetes.io/projected/0b42884e-e056-4bc3-85f7-d2879960b3fb-kube-api-access-bp8pl\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.405748 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.405544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b42884e-e056-4bc3-85f7-d2879960b3fb-tls-certs\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.405748 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:24.405633 2572 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 03 15:19:24.405748 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:24.405687 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b42884e-e056-4bc3-85f7-d2879960b3fb-tls-certs podName:0b42884e-e056-4bc3-85f7-d2879960b3fb nodeName:}" failed. No retries permitted until 2026-07-03 15:19:24.905670825 +0000 UTC m=+369.138103693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b42884e-e056-4bc3-85f7-d2879960b3fb-tls-certs") pod "kube-auth-proxy-9d497994-8fhn2" (UID: "0b42884e-e056-4bc3-85f7-d2879960b3fb") : secret "kube-auth-proxy-tls" not found Jul 03 15:19:24.407637 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.407605 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b42884e-e056-4bc3-85f7-d2879960b3fb-tmp\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.419196 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.419170 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8pl\" (UniqueName: \"kubernetes.io/projected/0b42884e-e056-4bc3-85f7-d2879960b3fb-kube-api-access-bp8pl\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.460206 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.460129 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvhs7"] Jul 03 15:19:24.463282 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.463267 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:24.466621 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.466600 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 03 15:19:24.477474 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.477452 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dg757\"" Jul 03 15:19:24.482924 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.482899 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvhs7"] Jul 03 15:19:24.506902 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.506874 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-cert\") pod \"odh-model-controller-7998c67dc4-hvhs7\" (UID: \"9ef602d3-50e5-4bc2-a02e-488c67bf14cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:24.507076 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.506912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq7r\" (UniqueName: \"kubernetes.io/projected/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-kube-api-access-dgq7r\") pod \"odh-model-controller-7998c67dc4-hvhs7\" (UID: \"9ef602d3-50e5-4bc2-a02e-488c67bf14cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:24.607519 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.607485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-cert\") pod \"odh-model-controller-7998c67dc4-hvhs7\" (UID: \"9ef602d3-50e5-4bc2-a02e-488c67bf14cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:24.607696 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.607544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq7r\" (UniqueName: \"kubernetes.io/projected/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-kube-api-access-dgq7r\") pod \"odh-model-controller-7998c67dc4-hvhs7\" (UID: \"9ef602d3-50e5-4bc2-a02e-488c67bf14cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:24.607696 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:24.607639 2572 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 03 15:19:24.607811 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:24.607723 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-cert podName:9ef602d3-50e5-4bc2-a02e-488c67bf14cf nodeName:}" failed. No retries permitted until 2026-07-03 15:19:25.107701903 +0000 UTC m=+369.340134793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-cert") pod "odh-model-controller-7998c67dc4-hvhs7" (UID: "9ef602d3-50e5-4bc2-a02e-488c67bf14cf") : secret "odh-model-controller-webhook-cert" not found Jul 03 15:19:24.621353 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.621324 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq7r\" (UniqueName: \"kubernetes.io/projected/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-kube-api-access-dgq7r\") pod \"odh-model-controller-7998c67dc4-hvhs7\" (UID: \"9ef602d3-50e5-4bc2-a02e-488c67bf14cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:24.911271 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.911234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b42884e-e056-4bc3-85f7-d2879960b3fb-tls-certs\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:24.913560 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:24.913533 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b42884e-e056-4bc3-85f7-d2879960b3fb-tls-certs\") pod \"kube-auth-proxy-9d497994-8fhn2\" (UID: \"0b42884e-e056-4bc3-85f7-d2879960b3fb\") " pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:25.113289 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:25.113254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-cert\") pod \"odh-model-controller-7998c67dc4-hvhs7\" (UID: \"9ef602d3-50e5-4bc2-a02e-488c67bf14cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:25.115714 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:25.115682 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ef602d3-50e5-4bc2-a02e-488c67bf14cf-cert\") pod \"odh-model-controller-7998c67dc4-hvhs7\" (UID: \"9ef602d3-50e5-4bc2-a02e-488c67bf14cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:25.125562 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:25.125538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" Jul 03 15:19:25.275732 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:25.275633 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9d497994-8fhn2"] Jul 03 15:19:25.277902 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:19:25.277870 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b42884e_e056_4bc3_85f7_d2879960b3fb.slice/crio-177ad4bde96fb1deb566a2eb58253b83b626ffa5584e24d716c3bffbcca32747 WatchSource:0}: Error finding container 177ad4bde96fb1deb566a2eb58253b83b626ffa5584e24d716c3bffbcca32747: Status 404 returned error can't find the container with id 177ad4bde96fb1deb566a2eb58253b83b626ffa5584e24d716c3bffbcca32747 Jul 03 15:19:25.374031 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:25.374002 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:25.729168 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:25.729079 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" event={"ID":"0b42884e-e056-4bc3-85f7-d2879960b3fb","Type":"ContainerStarted","Data":"177ad4bde96fb1deb566a2eb58253b83b626ffa5584e24d716c3bffbcca32747"} Jul 03 15:19:25.758591 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:25.758561 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvhs7"] Jul 03 15:19:25.762136 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:19:25.762103 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef602d3_50e5_4bc2_a02e_488c67bf14cf.slice/crio-6a1efe1e00fc860b3cb060833aaa5bfc03f7421d50bb15d8d649e79f735c68a6 WatchSource:0}: Error finding container 6a1efe1e00fc860b3cb060833aaa5bfc03f7421d50bb15d8d649e79f735c68a6: Status 404 returned error can't find the container with id 6a1efe1e00fc860b3cb060833aaa5bfc03f7421d50bb15d8d649e79f735c68a6 Jul 03 15:19:26.735844 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:26.735803 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" event={"ID":"9ef602d3-50e5-4bc2-a02e-488c67bf14cf","Type":"ContainerStarted","Data":"6a1efe1e00fc860b3cb060833aaa5bfc03f7421d50bb15d8d649e79f735c68a6"} Jul 03 15:19:29.751618 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:29.751579 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" event={"ID":"0b42884e-e056-4bc3-85f7-d2879960b3fb","Type":"ContainerStarted","Data":"13e4017ec0dfdf7bc5bc4c1636761bdf2233c6fb4a3e1184b8a9ea7b09ffbb83"} Jul 03 15:19:29.753259 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:29.753220 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" event={"ID":"9ef602d3-50e5-4bc2-a02e-488c67bf14cf","Type":"ContainerStarted","Data":"3e0ee74bbadf5c5193191d01691f1f197963065f59dc2f69caa9bf8ca2403b66"} Jul 03 15:19:29.753376 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:29.753347 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:29.768977 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:29.768924 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9d497994-8fhn2" podStartSLOduration=1.9792913300000001 podStartE2EDuration="5.768909439s" totalImagesPullingTime="3.789618109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:19:29.767024189 +0000 UTC m=+373.999457078" watchObservedRunningTime="2026-07-03 15:19:29.768909439 +0000 UTC m=+374.001342349" Jul 03 15:19:29.782914 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:29.782853 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" podStartSLOduration=2.477530754 podStartE2EDuration="5.782832155s" totalImagesPullingTime="3.305301401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:19:29.78224764 +0000 UTC m=+374.014680530" watchObservedRunningTime="2026-07-03 15:19:29.782832155 +0000 UTC m=+374.015265045" Jul 03 15:19:30.573870 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.573833 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj"] Jul 03 15:19:30.577377 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.577360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.580115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.580087 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tbcth\"" Jul 03 15:19:30.580115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.580115 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 03 15:19:30.589972 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.589944 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj"] Jul 03 15:19:30.659559 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659527 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659559 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbdb13f-19b4-4460-8655-616ce959226a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659636 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659666 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbdb13f-19b4-4460-8655-616ce959226a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6c9\" (UniqueName: \"kubernetes.io/projected/cfbdb13f-19b4-4460-8655-616ce959226a-kube-api-access-qj6c9\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659722 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbdb13f-19b4-4460-8655-616ce959226a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.659966 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.659800 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.760841 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.760806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.760841 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.760843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.760955 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbdb13f-19b4-4460-8655-616ce959226a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6c9\" (UniqueName: \"kubernetes.io/projected/cfbdb13f-19b4-4460-8655-616ce959226a-kube-api-access-qj6c9\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbdb13f-19b4-4460-8655-616ce959226a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761167 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761245 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761297 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbdb13f-19b4-4460-8655-616ce959226a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761666 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761507 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761666 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761655 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.761808 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.761784 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbdb13f-19b4-4460-8655-616ce959226a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.763524 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.763496 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbdb13f-19b4-4460-8655-616ce959226a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.763803 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.763785 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbdb13f-19b4-4460-8655-616ce959226a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.771309 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.771285 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6c9\" (UniqueName: \"kubernetes.io/projected/cfbdb13f-19b4-4460-8655-616ce959226a-kube-api-access-qj6c9\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.771410 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.771293 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbdb13f-19b4-4460-8655-616ce959226a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678gfsbj\" (UID: \"cfbdb13f-19b4-4460-8655-616ce959226a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.803517 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.803482 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs"] Jul 03 15:19:30.807871 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.807850 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.816722 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:30.816675 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-cert\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" Jul 03 15:19:30.816871 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:30.816763 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" Jul 03 15:19:30.816871 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:30.816794 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"lws-controller-manager-dockercfg-5fv4h\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5fv4h\"" type="*v1.Secret" Jul 03 15:19:30.816871 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:30.816811 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"lws-controller-manager-6c796b4fcc-d2pfs\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" podUID="938eb9a5-dd24-4749-8ecd-8ced9aec7901" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.817074 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:19:30.816868 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"lws-manager-config\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" Jul 03 15:19:30.848895 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.848860 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs"] Jul 03 15:19:30.862666 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.862638 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/938eb9a5-dd24-4749-8ecd-8ced9aec7901-cert\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.862666 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.862670 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/938eb9a5-dd24-4749-8ecd-8ced9aec7901-manager-config\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.862878 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.862733 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kkjt\" (UniqueName: \"kubernetes.io/projected/938eb9a5-dd24-4749-8ecd-8ced9aec7901-kube-api-access-8kkjt\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.862923 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.862902 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/938eb9a5-dd24-4749-8ecd-8ced9aec7901-metrics-cert\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.888700 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.888670 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:30.964109 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.964079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kkjt\" (UniqueName: \"kubernetes.io/projected/938eb9a5-dd24-4749-8ecd-8ced9aec7901-kube-api-access-8kkjt\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.964281 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.964119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/938eb9a5-dd24-4749-8ecd-8ced9aec7901-metrics-cert\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.964281 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.964240 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/938eb9a5-dd24-4749-8ecd-8ced9aec7901-cert\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.964281 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.964257 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/938eb9a5-dd24-4749-8ecd-8ced9aec7901-manager-config\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:30.976654 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:30.976627 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kkjt\" (UniqueName: \"kubernetes.io/projected/938eb9a5-dd24-4749-8ecd-8ced9aec7901-kube-api-access-8kkjt\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:31.035938 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.035916 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj"] Jul 03 15:19:31.038364 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:19:31.038342 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbdb13f_19b4_4460_8655_616ce959226a.slice/crio-d23383923dfebe114d78d559658c30bcbba7294a71d03b33889e084915108576 WatchSource:0}: Error finding container d23383923dfebe114d78d559658c30bcbba7294a71d03b33889e084915108576: Status 404 returned error can't find the container with id d23383923dfebe114d78d559658c30bcbba7294a71d03b33889e084915108576 Jul 03 15:19:31.678431 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.678401 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 03 15:19:31.687322 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.687285 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/938eb9a5-dd24-4749-8ecd-8ced9aec7901-metrics-cert\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:31.769665 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.769625 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" event={"ID":"cfbdb13f-19b4-4460-8655-616ce959226a","Type":"ContainerStarted","Data":"d23383923dfebe114d78d559658c30bcbba7294a71d03b33889e084915108576"} Jul 03 15:19:31.853408 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.853379 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 03 15:19:31.856090 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.856063 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/938eb9a5-dd24-4749-8ecd-8ced9aec7901-manager-config\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:31.921121 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.921087 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 03 15:19:31.927886 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.927856 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/938eb9a5-dd24-4749-8ecd-8ced9aec7901-cert\") pod \"lws-controller-manager-6c796b4fcc-d2pfs\" (UID: \"938eb9a5-dd24-4749-8ecd-8ced9aec7901\") " pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:31.958434 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:31.958346 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5fv4h\"" Jul 03 15:19:32.018064 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:32.018028 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:32.268564 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:32.268486 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs"] Jul 03 15:19:32.273652 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:19:32.273621 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938eb9a5_dd24_4749_8ecd_8ced9aec7901.slice/crio-7f4929a77d75ba5e9940693a3d4cb8425ee095986e4f20520c2df1cc9763389e WatchSource:0}: Error finding container 7f4929a77d75ba5e9940693a3d4cb8425ee095986e4f20520c2df1cc9763389e: Status 404 returned error can't find the container with id 7f4929a77d75ba5e9940693a3d4cb8425ee095986e4f20520c2df1cc9763389e Jul 03 15:19:32.780062 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:32.780026 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" event={"ID":"938eb9a5-dd24-4749-8ecd-8ced9aec7901","Type":"ContainerStarted","Data":"7f4929a77d75ba5e9940693a3d4cb8425ee095986e4f20520c2df1cc9763389e"} Jul 03 15:19:34.109983 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.109941 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 03 15:19:34.110290 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.110034 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 03 15:19:34.110290 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.110075 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 03 15:19:34.789876 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.789838 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" event={"ID":"938eb9a5-dd24-4749-8ecd-8ced9aec7901","Type":"ContainerStarted","Data":"d042a44cdbb8649d706f8cf80140e6f3203c959ad628afb5a9249abc3c8a34b7"} Jul 03 15:19:34.790161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.789907 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:19:34.791387 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.791360 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" event={"ID":"cfbdb13f-19b4-4460-8655-616ce959226a","Type":"ContainerStarted","Data":"dc7e0e66c962a286805f84b0ee74a257ad83df3240a1ed14f915f76dcf908ae7"} Jul 03 15:19:34.845345 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.845292 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" podStartSLOduration=3.011251572 podStartE2EDuration="4.845275398s" totalImagesPullingTime="1.834023826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:19:34.845214777 +0000 UTC m=+379.077647677" watchObservedRunningTime="2026-07-03 15:19:34.845275398 +0000 UTC m=+379.077708290" Jul 03 15:19:34.888851 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.888821 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:34.992865 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:34.992818 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" podStartSLOduration=1.9234261620000002 podStartE2EDuration="4.992799282s" totalImagesPullingTime="3.06937312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:19:34.992699939 +0000 UTC m=+379.225132829" watchObservedRunningTime="2026-07-03 15:19:34.992799282 +0000 UTC m=+379.225232170" Jul 03 15:19:35.893657 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:35.893625 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:36.801503 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:36.801472 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:36.802503 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:36.802475 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678gfsbj" Jul 03 15:19:40.759479 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:40.759451 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hvhs7" Jul 03 15:19:45.799350 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:19:45.799317 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c796b4fcc-d2pfs" Jul 03 15:20:37.091753 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.091708 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b"] Jul 03 15:20:37.095239 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.095220 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.098264 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.098238 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 03 15:20:37.098385 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.098275 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dp9v2\"" Jul 03 15:20:37.099264 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.099248 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 03 15:20:37.103716 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.103691 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b"] Jul 03 15:20:37.198251 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.198215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.198432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.198260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sckvr\" (UniqueName: \"kubernetes.io/projected/d383f0b3-a2ea-4807-ad91-22139105766d-kube-api-access-sckvr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.198432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.198359 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.299624 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.299591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.299748 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.299631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sckvr\" (UniqueName: \"kubernetes.io/projected/d383f0b3-a2ea-4807-ad91-22139105766d-kube-api-access-sckvr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.299748 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.299693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.300115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.300099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.300159 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.300105 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.308611 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.308592 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sckvr\" (UniqueName: \"kubernetes.io/projected/d383f0b3-a2ea-4807-ad91-22139105766d-kube-api-access-sckvr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.405430 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.405339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:37.537559 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.537530 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b"] Jul 03 15:20:37.540028 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:20:37.539999 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd383f0b3_a2ea_4807_ad91_22139105766d.slice/crio-6e14dbb747c976e2a32fa45be4ea6c42bb27bb1c5c6147bace71148965e712bd WatchSource:0}: Error finding container 6e14dbb747c976e2a32fa45be4ea6c42bb27bb1c5c6147bace71148965e712bd: Status 404 returned error can't find the container with id 6e14dbb747c976e2a32fa45be4ea6c42bb27bb1c5c6147bace71148965e712bd Jul 03 15:20:37.758786 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.758663 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf"] Jul 03 15:20:37.761840 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.761817 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:37.785381 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.785345 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf"] Jul 03 15:20:37.903426 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.903391 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:37.903613 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.903508 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:37.903613 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:37.903547 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57b2l\" (UniqueName: \"kubernetes.io/projected/bb354ebe-e866-470f-b7d7-aed008e1cee8-kube-api-access-57b2l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.004299 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.004265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.004436 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.004317 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57b2l\" (UniqueName: \"kubernetes.io/projected/bb354ebe-e866-470f-b7d7-aed008e1cee8-kube-api-access-57b2l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.004436 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.004341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.004750 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.004731 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.004787 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.004777 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.012920 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.012868 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57b2l\" (UniqueName: \"kubernetes.io/projected/bb354ebe-e866-470f-b7d7-aed008e1cee8-kube-api-access-57b2l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.051926 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.051891 2572 generic.go:356] "Generic (PLEG): container finished" podID="d383f0b3-a2ea-4807-ad91-22139105766d" containerID="b1436d0a8e2c6b87bb1ddbd234e26eed3954ee766e52520271802d8c0d8cd92c" exitCode=0 Jul 03 15:20:38.052090 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.051934 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" event={"ID":"d383f0b3-a2ea-4807-ad91-22139105766d","Type":"ContainerDied","Data":"b1436d0a8e2c6b87bb1ddbd234e26eed3954ee766e52520271802d8c0d8cd92c"} Jul 03 15:20:38.052090 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.051971 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" event={"ID":"d383f0b3-a2ea-4807-ad91-22139105766d","Type":"ContainerStarted","Data":"6e14dbb747c976e2a32fa45be4ea6c42bb27bb1c5c6147bace71148965e712bd"} Jul 03 15:20:38.075606 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.075576 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:38.205557 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.205533 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf"] Jul 03 15:20:38.207839 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:20:38.207813 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb354ebe_e866_470f_b7d7_aed008e1cee8.slice/crio-beb056aa4d249af55a4de3d271fb46b2300500a9a64e0e4fa0bc7215c39b0034 WatchSource:0}: Error finding container beb056aa4d249af55a4de3d271fb46b2300500a9a64e0e4fa0bc7215c39b0034: Status 404 returned error can't find the container with id beb056aa4d249af55a4de3d271fb46b2300500a9a64e0e4fa0bc7215c39b0034 Jul 03 15:20:38.292769 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.292733 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs"] Jul 03 15:20:38.296226 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.296209 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.306207 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.306179 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs"] Jul 03 15:20:38.408602 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.408559 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflxq\" (UniqueName: \"kubernetes.io/projected/1978064d-027f-4af4-947a-75c8e46f7e98-kube-api-access-rflxq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.408602 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.408607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.408821 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.408653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.509352 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.509320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.509527 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.509400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflxq\" (UniqueName: \"kubernetes.io/projected/1978064d-027f-4af4-947a-75c8e46f7e98-kube-api-access-rflxq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.509527 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.509425 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.509715 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.509692 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.509759 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.509729 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.519163 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.519091 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflxq\" (UniqueName: \"kubernetes.io/projected/1978064d-027f-4af4-947a-75c8e46f7e98-kube-api-access-rflxq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.606391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.606348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:38.747270 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.747198 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs"] Jul 03 15:20:38.749299 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:20:38.749272 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1978064d_027f_4af4_947a_75c8e46f7e98.slice/crio-31645eda130f29a58083c4d7828e58175ba1b48d45af9d9c1bb19d63fe83cd54 WatchSource:0}: Error finding container 31645eda130f29a58083c4d7828e58175ba1b48d45af9d9c1bb19d63fe83cd54: Status 404 returned error can't find the container with id 31645eda130f29a58083c4d7828e58175ba1b48d45af9d9c1bb19d63fe83cd54 Jul 03 15:20:38.903062 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.903030 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj"] Jul 03 15:20:38.906368 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.906349 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:38.915905 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:38.915875 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj"] Jul 03 15:20:39.014714 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.014679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2lhx\" (UniqueName: \"kubernetes.io/projected/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-kube-api-access-l2lhx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.014899 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.014725 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.014899 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.014757 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.057584 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.057504 2572 generic.go:356] "Generic (PLEG): container finished" podID="d383f0b3-a2ea-4807-ad91-22139105766d" containerID="19c0750c9357796d085b02dac540ddfccb5e674d2e2cac81a790954f7dc66e4a" exitCode=0 Jul 03 15:20:39.057584 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.057559 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" event={"ID":"d383f0b3-a2ea-4807-ad91-22139105766d","Type":"ContainerDied","Data":"19c0750c9357796d085b02dac540ddfccb5e674d2e2cac81a790954f7dc66e4a"} Jul 03 15:20:39.059199 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.059163 2572 generic.go:356] "Generic (PLEG): container finished" podID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerID="5ef3d33a7dc3778d56c82184a3dfd4efa264657fb2d6bc75cfe701e77ea24c82" exitCode=0 Jul 03 15:20:39.059260 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.059243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" event={"ID":"bb354ebe-e866-470f-b7d7-aed008e1cee8","Type":"ContainerDied","Data":"5ef3d33a7dc3778d56c82184a3dfd4efa264657fb2d6bc75cfe701e77ea24c82"} Jul 03 15:20:39.059307 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.059282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" event={"ID":"bb354ebe-e866-470f-b7d7-aed008e1cee8","Type":"ContainerStarted","Data":"beb056aa4d249af55a4de3d271fb46b2300500a9a64e0e4fa0bc7215c39b0034"} Jul 03 15:20:39.060811 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.060790 2572 generic.go:356] "Generic (PLEG): container finished" podID="1978064d-027f-4af4-947a-75c8e46f7e98" containerID="0da3b532bb8881d4006df2e255153ddf60c76af871f4cf646cc8c7c593c0a720" exitCode=0 Jul 03 15:20:39.060886 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.060823 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" event={"ID":"1978064d-027f-4af4-947a-75c8e46f7e98","Type":"ContainerDied","Data":"0da3b532bb8881d4006df2e255153ddf60c76af871f4cf646cc8c7c593c0a720"} Jul 03 15:20:39.060886 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.060848 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" event={"ID":"1978064d-027f-4af4-947a-75c8e46f7e98","Type":"ContainerStarted","Data":"31645eda130f29a58083c4d7828e58175ba1b48d45af9d9c1bb19d63fe83cd54"} Jul 03 15:20:39.115202 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.115170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2lhx\" (UniqueName: \"kubernetes.io/projected/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-kube-api-access-l2lhx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.115416 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.115226 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.116120 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.115437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.116120 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.115750 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.116120 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.115755 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.124078 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.124054 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2lhx\" (UniqueName: \"kubernetes.io/projected/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-kube-api-access-l2lhx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.256871 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.256831 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:39.393463 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:39.393438 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj"] Jul 03 15:20:39.395685 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:20:39.395659 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97f5a4b_c499_46e7_a37e_76683bf3aa6a.slice/crio-ab527d65dbf73b23edcff3ec14efa7cdee08612812fb25c5644aefd57e213eb1 WatchSource:0}: Error finding container ab527d65dbf73b23edcff3ec14efa7cdee08612812fb25c5644aefd57e213eb1: Status 404 returned error can't find the container with id ab527d65dbf73b23edcff3ec14efa7cdee08612812fb25c5644aefd57e213eb1 Jul 03 15:20:40.073163 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:40.073088 2572 generic.go:356] "Generic (PLEG): container finished" podID="d383f0b3-a2ea-4807-ad91-22139105766d" containerID="2e29e9c1fd04b307ac192b5a140aafdf457e6bf6c1dfa23deb847e60cb829378" exitCode=0 Jul 03 15:20:40.073329 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:40.073180 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" event={"ID":"d383f0b3-a2ea-4807-ad91-22139105766d","Type":"ContainerDied","Data":"2e29e9c1fd04b307ac192b5a140aafdf457e6bf6c1dfa23deb847e60cb829378"} Jul 03 15:20:40.074933 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:40.074911 2572 generic.go:356] "Generic (PLEG): container finished" podID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerID="c5fae32c2062a9619119c652dcfa3b84f344bb38d6f3d61c5f0577b232db46d9" exitCode=0 Jul 03 15:20:40.075052 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:40.074977 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" event={"ID":"bb354ebe-e866-470f-b7d7-aed008e1cee8","Type":"ContainerDied","Data":"c5fae32c2062a9619119c652dcfa3b84f344bb38d6f3d61c5f0577b232db46d9"} Jul 03 15:20:40.076477 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:40.076453 2572 generic.go:356] "Generic (PLEG): container finished" podID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerID="ce518e05ad3104d687ea542f0d0c6dc2b58face4a3d1a49f5f293b747224cf67" exitCode=0 Jul 03 15:20:40.076606 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:40.076504 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" event={"ID":"e97f5a4b-c499-46e7-a37e-76683bf3aa6a","Type":"ContainerDied","Data":"ce518e05ad3104d687ea542f0d0c6dc2b58face4a3d1a49f5f293b747224cf67"} Jul 03 15:20:40.076606 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:40.076539 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" event={"ID":"e97f5a4b-c499-46e7-a37e-76683bf3aa6a","Type":"ContainerStarted","Data":"ab527d65dbf73b23edcff3ec14efa7cdee08612812fb25c5644aefd57e213eb1"} Jul 03 15:20:41.083748 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.083715 2572 generic.go:356] "Generic (PLEG): container finished" podID="1978064d-027f-4af4-947a-75c8e46f7e98" containerID="b1c87b13aa3527a700201457e8ae3e34193c3f87b30fdcebbe04a351d00eafe2" exitCode=0 Jul 03 15:20:41.084127 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.083754 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" event={"ID":"1978064d-027f-4af4-947a-75c8e46f7e98","Type":"ContainerDied","Data":"b1c87b13aa3527a700201457e8ae3e34193c3f87b30fdcebbe04a351d00eafe2"} Jul 03 15:20:41.086230 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.086205 2572 generic.go:356] "Generic (PLEG): container finished" podID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerID="a6a84dfeb3ecb3562fb19df33ff6bf0dc44f3fb7668c85069ac35cdd029df893" exitCode=0 Jul 03 15:20:41.086326 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.086265 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" event={"ID":"bb354ebe-e866-470f-b7d7-aed008e1cee8","Type":"ContainerDied","Data":"a6a84dfeb3ecb3562fb19df33ff6bf0dc44f3fb7668c85069ac35cdd029df893"} Jul 03 15:20:41.227168 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.227143 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:41.333569 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.333535 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sckvr\" (UniqueName: \"kubernetes.io/projected/d383f0b3-a2ea-4807-ad91-22139105766d-kube-api-access-sckvr\") pod \"d383f0b3-a2ea-4807-ad91-22139105766d\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " Jul 03 15:20:41.333746 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.333579 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-util\") pod \"d383f0b3-a2ea-4807-ad91-22139105766d\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " Jul 03 15:20:41.333746 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.333629 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-bundle\") pod \"d383f0b3-a2ea-4807-ad91-22139105766d\" (UID: \"d383f0b3-a2ea-4807-ad91-22139105766d\") " Jul 03 15:20:41.334320 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.334290 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-bundle" (OuterVolumeSpecName: "bundle") pod "d383f0b3-a2ea-4807-ad91-22139105766d" (UID: "d383f0b3-a2ea-4807-ad91-22139105766d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:41.335866 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.335842 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d383f0b3-a2ea-4807-ad91-22139105766d-kube-api-access-sckvr" (OuterVolumeSpecName: "kube-api-access-sckvr") pod "d383f0b3-a2ea-4807-ad91-22139105766d" (UID: "d383f0b3-a2ea-4807-ad91-22139105766d"). InnerVolumeSpecName "kube-api-access-sckvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:20:41.339196 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.339170 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-util" (OuterVolumeSpecName: "util") pod "d383f0b3-a2ea-4807-ad91-22139105766d" (UID: "d383f0b3-a2ea-4807-ad91-22139105766d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:41.434740 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.434695 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sckvr\" (UniqueName: \"kubernetes.io/projected/d383f0b3-a2ea-4807-ad91-22139105766d-kube-api-access-sckvr\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:41.434740 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.434730 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-util\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:41.434740 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:41.434742 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d383f0b3-a2ea-4807-ad91-22139105766d-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:42.093084 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.093049 2572 generic.go:356] "Generic (PLEG): container finished" podID="1978064d-027f-4af4-947a-75c8e46f7e98" containerID="d99acba7479f1c733cfc218de809bf9fdbb648ac9752b2890fcab85f65b1064f" exitCode=0 Jul 03 15:20:42.093465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.093132 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" event={"ID":"1978064d-027f-4af4-947a-75c8e46f7e98","Type":"ContainerDied","Data":"d99acba7479f1c733cfc218de809bf9fdbb648ac9752b2890fcab85f65b1064f"} Jul 03 15:20:42.095045 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.095018 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" event={"ID":"d383f0b3-a2ea-4807-ad91-22139105766d","Type":"ContainerDied","Data":"6e14dbb747c976e2a32fa45be4ea6c42bb27bb1c5c6147bace71148965e712bd"} Jul 03 15:20:42.095045 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.095037 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b" Jul 03 15:20:42.095207 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.095064 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e14dbb747c976e2a32fa45be4ea6c42bb27bb1c5c6147bace71148965e712bd" Jul 03 15:20:42.096747 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.096724 2572 generic.go:356] "Generic (PLEG): container finished" podID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerID="c3abbb1848dba6b9fd7881f956c1b66598be254dc379c9bcc245d5b3602a014b" exitCode=0 Jul 03 15:20:42.096877 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.096805 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" event={"ID":"e97f5a4b-c499-46e7-a37e-76683bf3aa6a","Type":"ContainerDied","Data":"c3abbb1848dba6b9fd7881f956c1b66598be254dc379c9bcc245d5b3602a014b"} Jul 03 15:20:42.232828 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.232800 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:42.340414 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.340382 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57b2l\" (UniqueName: \"kubernetes.io/projected/bb354ebe-e866-470f-b7d7-aed008e1cee8-kube-api-access-57b2l\") pod \"bb354ebe-e866-470f-b7d7-aed008e1cee8\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " Jul 03 15:20:42.340571 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.340475 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-util\") pod \"bb354ebe-e866-470f-b7d7-aed008e1cee8\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " Jul 03 15:20:42.340571 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.340510 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-bundle\") pod \"bb354ebe-e866-470f-b7d7-aed008e1cee8\" (UID: \"bb354ebe-e866-470f-b7d7-aed008e1cee8\") " Jul 03 15:20:42.341045 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.341018 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-bundle" (OuterVolumeSpecName: "bundle") pod "bb354ebe-e866-470f-b7d7-aed008e1cee8" (UID: "bb354ebe-e866-470f-b7d7-aed008e1cee8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:42.342513 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.342487 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb354ebe-e866-470f-b7d7-aed008e1cee8-kube-api-access-57b2l" (OuterVolumeSpecName: "kube-api-access-57b2l") pod "bb354ebe-e866-470f-b7d7-aed008e1cee8" (UID: "bb354ebe-e866-470f-b7d7-aed008e1cee8"). InnerVolumeSpecName "kube-api-access-57b2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:20:42.346553 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.346412 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-util" (OuterVolumeSpecName: "util") pod "bb354ebe-e866-470f-b7d7-aed008e1cee8" (UID: "bb354ebe-e866-470f-b7d7-aed008e1cee8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:42.442055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.441979 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:42.442055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.442048 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57b2l\" (UniqueName: \"kubernetes.io/projected/bb354ebe-e866-470f-b7d7-aed008e1cee8-kube-api-access-57b2l\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:42.442055 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:42.442060 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb354ebe-e866-470f-b7d7-aed008e1cee8-util\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:43.102770 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.102746 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" Jul 03 15:20:43.103201 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.102740 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf" event={"ID":"bb354ebe-e866-470f-b7d7-aed008e1cee8","Type":"ContainerDied","Data":"beb056aa4d249af55a4de3d271fb46b2300500a9a64e0e4fa0bc7215c39b0034"} Jul 03 15:20:43.103201 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.102856 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beb056aa4d249af55a4de3d271fb46b2300500a9a64e0e4fa0bc7215c39b0034" Jul 03 15:20:43.104738 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.104711 2572 generic.go:356] "Generic (PLEG): container finished" podID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerID="44f8d50ea9c93248790eea736231a7153e808ff6523aca7559d44be3045eed61" exitCode=0 Jul 03 15:20:43.104870 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.104798 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" event={"ID":"e97f5a4b-c499-46e7-a37e-76683bf3aa6a","Type":"ContainerDied","Data":"44f8d50ea9c93248790eea736231a7153e808ff6523aca7559d44be3045eed61"} Jul 03 15:20:43.230677 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.230652 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:43.353049 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.352942 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-util\") pod \"1978064d-027f-4af4-947a-75c8e46f7e98\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " Jul 03 15:20:43.353049 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.353049 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rflxq\" (UniqueName: \"kubernetes.io/projected/1978064d-027f-4af4-947a-75c8e46f7e98-kube-api-access-rflxq\") pod \"1978064d-027f-4af4-947a-75c8e46f7e98\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " Jul 03 15:20:43.353266 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.353097 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-bundle\") pod \"1978064d-027f-4af4-947a-75c8e46f7e98\" (UID: \"1978064d-027f-4af4-947a-75c8e46f7e98\") " Jul 03 15:20:43.353691 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.353654 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-bundle" (OuterVolumeSpecName: "bundle") pod "1978064d-027f-4af4-947a-75c8e46f7e98" (UID: "1978064d-027f-4af4-947a-75c8e46f7e98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:43.355265 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.355240 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1978064d-027f-4af4-947a-75c8e46f7e98-kube-api-access-rflxq" (OuterVolumeSpecName: "kube-api-access-rflxq") pod "1978064d-027f-4af4-947a-75c8e46f7e98" (UID: "1978064d-027f-4af4-947a-75c8e46f7e98"). InnerVolumeSpecName "kube-api-access-rflxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:20:43.358594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.358565 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-util" (OuterVolumeSpecName: "util") pod "1978064d-027f-4af4-947a-75c8e46f7e98" (UID: "1978064d-027f-4af4-947a-75c8e46f7e98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:43.454626 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.454571 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflxq\" (UniqueName: \"kubernetes.io/projected/1978064d-027f-4af4-947a-75c8e46f7e98-kube-api-access-rflxq\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:43.454626 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.454615 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:43.454626 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:43.454626 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1978064d-027f-4af4-947a-75c8e46f7e98-util\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:44.111643 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.111611 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" Jul 03 15:20:44.112045 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.111634 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs" event={"ID":"1978064d-027f-4af4-947a-75c8e46f7e98","Type":"ContainerDied","Data":"31645eda130f29a58083c4d7828e58175ba1b48d45af9d9c1bb19d63fe83cd54"} Jul 03 15:20:44.112045 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.111717 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31645eda130f29a58083c4d7828e58175ba1b48d45af9d9c1bb19d63fe83cd54" Jul 03 15:20:44.237452 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.237424 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:44.364290 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.364196 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-util\") pod \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " Jul 03 15:20:44.364452 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.364292 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2lhx\" (UniqueName: \"kubernetes.io/projected/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-kube-api-access-l2lhx\") pod \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " Jul 03 15:20:44.364452 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.364318 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-bundle\") pod \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\" (UID: \"e97f5a4b-c499-46e7-a37e-76683bf3aa6a\") " Jul 03 15:20:44.364833 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.364807 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-bundle" (OuterVolumeSpecName: "bundle") pod "e97f5a4b-c499-46e7-a37e-76683bf3aa6a" (UID: "e97f5a4b-c499-46e7-a37e-76683bf3aa6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:44.366486 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.366456 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-kube-api-access-l2lhx" (OuterVolumeSpecName: "kube-api-access-l2lhx") pod "e97f5a4b-c499-46e7-a37e-76683bf3aa6a" (UID: "e97f5a4b-c499-46e7-a37e-76683bf3aa6a"). InnerVolumeSpecName "kube-api-access-l2lhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:20:44.369809 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.369782 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-util" (OuterVolumeSpecName: "util") pod "e97f5a4b-c499-46e7-a37e-76683bf3aa6a" (UID: "e97f5a4b-c499-46e7-a37e-76683bf3aa6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:20:44.465709 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.465666 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2lhx\" (UniqueName: \"kubernetes.io/projected/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-kube-api-access-l2lhx\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:44.465709 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.465697 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:44.465709 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:44.465707 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e97f5a4b-c499-46e7-a37e-76683bf3aa6a-util\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:20:45.117515 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:45.117472 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" event={"ID":"e97f5a4b-c499-46e7-a37e-76683bf3aa6a","Type":"ContainerDied","Data":"ab527d65dbf73b23edcff3ec14efa7cdee08612812fb25c5644aefd57e213eb1"} Jul 03 15:20:45.117515 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:45.117516 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj" Jul 03 15:20:45.117963 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:20:45.117530 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab527d65dbf73b23edcff3ec14efa7cdee08612812fb25c5644aefd57e213eb1" Jul 03 15:22:14.299727 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.299666 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qkdvb"] Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.299964 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d383f0b3-a2ea-4807-ad91-22139105766d" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.299978 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d383f0b3-a2ea-4807-ad91-22139105766d" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300006 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300015 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300033 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d383f0b3-a2ea-4807-ad91-22139105766d" containerName="util" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300041 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d383f0b3-a2ea-4807-ad91-22139105766d" containerName="util" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300051 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d383f0b3-a2ea-4807-ad91-22139105766d" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300059 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d383f0b3-a2ea-4807-ad91-22139105766d" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300069 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300077 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300091 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerName="util" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300097 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerName="util" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300103 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1978064d-027f-4af4-947a-75c8e46f7e98" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300108 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1978064d-027f-4af4-947a-75c8e46f7e98" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300117 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300122 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerName="extract" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300133 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300140 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300152 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1978064d-027f-4af4-947a-75c8e46f7e98" containerName="util" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300157 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1978064d-027f-4af4-947a-75c8e46f7e98" containerName="util" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300166 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1978064d-027f-4af4-947a-75c8e46f7e98" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300173 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1978064d-027f-4af4-947a-75c8e46f7e98" containerName="pull" Jul 03 15:22:14.300175 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300187 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerName="util" Jul 03 15:22:14.300959 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300195 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerName="util" Jul 03 15:22:14.300959 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300266 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb354ebe-e866-470f-b7d7-aed008e1cee8" containerName="extract" Jul 03 15:22:14.300959 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300274 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97f5a4b-c499-46e7-a37e-76683bf3aa6a" containerName="extract" Jul 03 15:22:14.300959 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300280 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1978064d-027f-4af4-947a-75c8e46f7e98" containerName="extract" Jul 03 15:22:14.300959 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.300288 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d383f0b3-a2ea-4807-ad91-22139105766d" containerName="extract" Jul 03 15:22:14.303121 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.303105 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qkdvb" Jul 03 15:22:14.305862 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.305834 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 03 15:22:14.306021 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.305933 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 03 15:22:14.306021 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.306008 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q7cf6\"" Jul 03 15:22:14.310750 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.310724 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qkdvb"] Jul 03 15:22:14.381050 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.381019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jbc\" (UniqueName: \"kubernetes.io/projected/bb0cf2b2-2400-4dc5-9e36-9746a9dc935f-kube-api-access-g9jbc\") pod \"authorino-58cd955675-qkdvb\" (UID: \"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f\") " pod="kuadrant-system/authorino-58cd955675-qkdvb" Jul 03 15:22:14.481687 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.481652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jbc\" (UniqueName: \"kubernetes.io/projected/bb0cf2b2-2400-4dc5-9e36-9746a9dc935f-kube-api-access-g9jbc\") pod \"authorino-58cd955675-qkdvb\" (UID: \"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f\") " pod="kuadrant-system/authorino-58cd955675-qkdvb" Jul 03 15:22:14.493835 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.493798 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jbc\" (UniqueName: \"kubernetes.io/projected/bb0cf2b2-2400-4dc5-9e36-9746a9dc935f-kube-api-access-g9jbc\") pod \"authorino-58cd955675-qkdvb\" (UID: \"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f\") " pod="kuadrant-system/authorino-58cd955675-qkdvb" Jul 03 15:22:14.502059 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.502030 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-m5474"] Jul 03 15:22:14.505195 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.505179 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-m5474" Jul 03 15:22:14.511704 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.511679 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-m5474"] Jul 03 15:22:14.583051 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.583017 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk28q\" (UniqueName: \"kubernetes.io/projected/dcf1028d-4ab3-49e5-8df5-5f4b602f90d9-kube-api-access-dk28q\") pod \"authorino-75d556bd77-m5474\" (UID: \"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9\") " pod="kuadrant-system/authorino-75d556bd77-m5474" Jul 03 15:22:14.613731 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.613705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qkdvb" Jul 03 15:22:14.683968 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.683937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk28q\" (UniqueName: \"kubernetes.io/projected/dcf1028d-4ab3-49e5-8df5-5f4b602f90d9-kube-api-access-dk28q\") pod \"authorino-75d556bd77-m5474\" (UID: \"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9\") " pod="kuadrant-system/authorino-75d556bd77-m5474" Jul 03 15:22:14.695116 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.695086 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk28q\" (UniqueName: \"kubernetes.io/projected/dcf1028d-4ab3-49e5-8df5-5f4b602f90d9-kube-api-access-dk28q\") pod \"authorino-75d556bd77-m5474\" (UID: \"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9\") " pod="kuadrant-system/authorino-75d556bd77-m5474" Jul 03 15:22:14.745129 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.745101 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qkdvb"] Jul 03 15:22:14.747268 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:22:14.747241 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb0cf2b2_2400_4dc5_9e36_9746a9dc935f.slice/crio-43bc63cf941c7a8d9168c44134d61e2283d6ae8ec063f62f5cbe9379fb2b291d WatchSource:0}: Error finding container 43bc63cf941c7a8d9168c44134d61e2283d6ae8ec063f62f5cbe9379fb2b291d: Status 404 returned error can't find the container with id 43bc63cf941c7a8d9168c44134d61e2283d6ae8ec063f62f5cbe9379fb2b291d Jul 03 15:22:14.815255 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.815217 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-m5474" Jul 03 15:22:14.944602 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:14.944566 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-m5474"] Jul 03 15:22:14.947372 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:22:14.947340 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcf1028d_4ab3_49e5_8df5_5f4b602f90d9.slice/crio-846be86d979b59167700b8dddd8388426cc69a3a58e836db4069d13de127f1e8 WatchSource:0}: Error finding container 846be86d979b59167700b8dddd8388426cc69a3a58e836db4069d13de127f1e8: Status 404 returned error can't find the container with id 846be86d979b59167700b8dddd8388426cc69a3a58e836db4069d13de127f1e8 Jul 03 15:22:15.527611 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:15.527571 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-m5474" event={"ID":"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9","Type":"ContainerStarted","Data":"846be86d979b59167700b8dddd8388426cc69a3a58e836db4069d13de127f1e8"} Jul 03 15:22:15.529033 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:15.529002 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qkdvb" event={"ID":"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f","Type":"ContainerStarted","Data":"43bc63cf941c7a8d9168c44134d61e2283d6ae8ec063f62f5cbe9379fb2b291d"} Jul 03 15:22:18.547064 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:18.547028 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-m5474" event={"ID":"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9","Type":"ContainerStarted","Data":"6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4"} Jul 03 15:22:18.548418 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:18.548390 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qkdvb" event={"ID":"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f","Type":"ContainerStarted","Data":"48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58"} Jul 03 15:22:18.564004 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:18.563941 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-m5474" podStartSLOduration=1.156551431 podStartE2EDuration="4.563925984s" totalImagesPullingTime="3.407374553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:22:18.563354359 +0000 UTC m=+542.795787249" watchObservedRunningTime="2026-07-03 15:22:18.563925984 +0000 UTC m=+542.796358872" Jul 03 15:22:18.590164 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:18.590113 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qkdvb" podStartSLOduration=0.993514088 podStartE2EDuration="4.590096567s" totalImagesPullingTime="3.596582479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:22:18.588865419 +0000 UTC m=+542.821298311" watchObservedRunningTime="2026-07-03 15:22:18.590096567 +0000 UTC m=+542.822529456" Jul 03 15:22:18.596117 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:18.596022 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qkdvb"] Jul 03 15:22:20.557444 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:20.557407 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qkdvb" podUID="bb0cf2b2-2400-4dc5-9e36-9746a9dc935f" containerName="authorino" containerID="cri-o://48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58" gracePeriod=30 Jul 03 15:22:20.800707 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:20.800682 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qkdvb" Jul 03 15:22:20.834166 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:20.834142 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9jbc\" (UniqueName: \"kubernetes.io/projected/bb0cf2b2-2400-4dc5-9e36-9746a9dc935f-kube-api-access-g9jbc\") pod \"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f\" (UID: \"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f\") " Jul 03 15:22:20.836125 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:20.836099 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0cf2b2-2400-4dc5-9e36-9746a9dc935f-kube-api-access-g9jbc" (OuterVolumeSpecName: "kube-api-access-g9jbc") pod "bb0cf2b2-2400-4dc5-9e36-9746a9dc935f" (UID: "bb0cf2b2-2400-4dc5-9e36-9746a9dc935f"). InnerVolumeSpecName "kube-api-access-g9jbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:22:20.934781 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:20.934742 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jbc\" (UniqueName: \"kubernetes.io/projected/bb0cf2b2-2400-4dc5-9e36-9746a9dc935f-kube-api-access-g9jbc\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:22:21.563336 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.563299 2572 generic.go:356] "Generic (PLEG): container finished" podID="bb0cf2b2-2400-4dc5-9e36-9746a9dc935f" containerID="48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58" exitCode=0 Jul 03 15:22:21.563812 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.563347 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qkdvb" Jul 03 15:22:21.563812 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.563395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qkdvb" event={"ID":"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f","Type":"ContainerDied","Data":"48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58"} Jul 03 15:22:21.563812 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.563437 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qkdvb" event={"ID":"bb0cf2b2-2400-4dc5-9e36-9746a9dc935f","Type":"ContainerDied","Data":"43bc63cf941c7a8d9168c44134d61e2283d6ae8ec063f62f5cbe9379fb2b291d"} Jul 03 15:22:21.563812 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.563454 2572 scope.go:117] "RemoveContainer" containerID="48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58" Jul 03 15:22:21.572267 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.572247 2572 scope.go:117] "RemoveContainer" containerID="48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58" Jul 03 15:22:21.572546 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:22:21.572529 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58\": container with ID starting with 48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58 not found: ID does not exist" containerID="48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58" Jul 03 15:22:21.572593 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.572556 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58"} err="failed to get container status \"48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58\": rpc error: code = NotFound desc = could not find container \"48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58\": container with ID starting with 48f0089f37151d0dfa12b0e7fd10d45241bdf97c78e3ba546c90d85712856a58 not found: ID does not exist" Jul 03 15:22:21.583690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.583664 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qkdvb"] Jul 03 15:22:21.587313 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:21.587289 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qkdvb"] Jul 03 15:22:22.231075 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:22.231037 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb0cf2b2-2400-4dc5-9e36-9746a9dc935f" path="/var/lib/kubelet/pods/bb0cf2b2-2400-4dc5-9e36-9746a9dc935f/volumes" Jul 03 15:22:44.449057 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.449025 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-56q79"] Jul 03 15:22:44.449536 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.449298 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0cf2b2-2400-4dc5-9e36-9746a9dc935f" containerName="authorino" Jul 03 15:22:44.449536 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.449308 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0cf2b2-2400-4dc5-9e36-9746a9dc935f" containerName="authorino" Jul 03 15:22:44.449536 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.449359 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0cf2b2-2400-4dc5-9e36-9746a9dc935f" containerName="authorino" Jul 03 15:22:44.452563 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.452545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-56q79" Jul 03 15:22:44.461704 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.461676 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-56q79"] Jul 03 15:22:44.540169 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.540125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25mt\" (UniqueName: \"kubernetes.io/projected/46259325-3c58-434d-a380-4271b96a7e15-kube-api-access-s25mt\") pod \"authorino-5d4b67c446-56q79\" (UID: \"46259325-3c58-434d-a380-4271b96a7e15\") " pod="kuadrant-system/authorino-5d4b67c446-56q79" Jul 03 15:22:44.626396 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.626365 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-56q79"] Jul 03 15:22:44.626613 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:22:44.626595 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s25mt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-56q79" podUID="46259325-3c58-434d-a380-4271b96a7e15" Jul 03 15:22:44.640868 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.640839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25mt\" (UniqueName: \"kubernetes.io/projected/46259325-3c58-434d-a380-4271b96a7e15-kube-api-access-s25mt\") pod \"authorino-5d4b67c446-56q79\" (UID: \"46259325-3c58-434d-a380-4271b96a7e15\") " pod="kuadrant-system/authorino-5d4b67c446-56q79" Jul 03 15:22:44.657302 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.657271 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25mt\" (UniqueName: \"kubernetes.io/projected/46259325-3c58-434d-a380-4271b96a7e15-kube-api-access-s25mt\") pod \"authorino-5d4b67c446-56q79\" (UID: \"46259325-3c58-434d-a380-4271b96a7e15\") " pod="kuadrant-system/authorino-5d4b67c446-56q79" Jul 03 15:22:44.666553 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.666529 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-56q79" Jul 03 15:22:44.672101 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.672082 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-56q79" Jul 03 15:22:44.741598 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.741504 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s25mt\" (UniqueName: \"kubernetes.io/projected/46259325-3c58-434d-a380-4271b96a7e15-kube-api-access-s25mt\") pod \"46259325-3c58-434d-a380-4271b96a7e15\" (UID: \"46259325-3c58-434d-a380-4271b96a7e15\") " Jul 03 15:22:44.743653 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.743625 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46259325-3c58-434d-a380-4271b96a7e15-kube-api-access-s25mt" (OuterVolumeSpecName: "kube-api-access-s25mt") pod "46259325-3c58-434d-a380-4271b96a7e15" (UID: "46259325-3c58-434d-a380-4271b96a7e15"). InnerVolumeSpecName "kube-api-access-s25mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:22:44.842425 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:44.842384 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25mt\" (UniqueName: \"kubernetes.io/projected/46259325-3c58-434d-a380-4271b96a7e15-kube-api-access-s25mt\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:22:45.670801 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:45.670772 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-56q79" Jul 03 15:22:45.698782 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:45.698754 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-56q79"] Jul 03 15:22:45.704248 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:45.704221 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-56q79"] Jul 03 15:22:46.229907 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:46.229878 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46259325-3c58-434d-a380-4271b96a7e15" path="/var/lib/kubelet/pods/46259325-3c58-434d-a380-4271b96a7e15/volumes" Jul 03 15:22:48.844377 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:48.844348 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-m5474"] Jul 03 15:22:48.844774 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:48.844511 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-m5474" podUID="dcf1028d-4ab3-49e5-8df5-5f4b602f90d9" containerName="authorino" containerID="cri-o://6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4" gracePeriod=30 Jul 03 15:22:49.089639 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.089616 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-m5474" Jul 03 15:22:49.176459 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.176371 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk28q\" (UniqueName: \"kubernetes.io/projected/dcf1028d-4ab3-49e5-8df5-5f4b602f90d9-kube-api-access-dk28q\") pod \"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9\" (UID: \"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9\") " Jul 03 15:22:49.178393 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.178362 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf1028d-4ab3-49e5-8df5-5f4b602f90d9-kube-api-access-dk28q" (OuterVolumeSpecName: "kube-api-access-dk28q") pod "dcf1028d-4ab3-49e5-8df5-5f4b602f90d9" (UID: "dcf1028d-4ab3-49e5-8df5-5f4b602f90d9"). InnerVolumeSpecName "kube-api-access-dk28q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:22:49.277500 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.277465 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk28q\" (UniqueName: \"kubernetes.io/projected/dcf1028d-4ab3-49e5-8df5-5f4b602f90d9-kube-api-access-dk28q\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:22:49.689933 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.689898 2572 generic.go:356] "Generic (PLEG): container finished" podID="dcf1028d-4ab3-49e5-8df5-5f4b602f90d9" containerID="6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4" exitCode=0 Jul 03 15:22:49.690115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.689944 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-m5474" Jul 03 15:22:49.690115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.689980 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-m5474" event={"ID":"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9","Type":"ContainerDied","Data":"6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4"} Jul 03 15:22:49.690115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.690041 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-m5474" event={"ID":"dcf1028d-4ab3-49e5-8df5-5f4b602f90d9","Type":"ContainerDied","Data":"846be86d979b59167700b8dddd8388426cc69a3a58e836db4069d13de127f1e8"} Jul 03 15:22:49.690115 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.690058 2572 scope.go:117] "RemoveContainer" containerID="6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4" Jul 03 15:22:49.699129 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.699109 2572 scope.go:117] "RemoveContainer" containerID="6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4" Jul 03 15:22:49.699408 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:22:49.699388 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4\": container with ID starting with 6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4 not found: ID does not exist" containerID="6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4" Jul 03 15:22:49.699451 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.699418 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4"} err="failed to get container status \"6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4\": rpc error: code = NotFound desc = could not find container \"6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4\": container with ID starting with 6aade17682b13abe6214b44ced557559481d1efece9666f50d1cd878e670e5a4 not found: ID does not exist" Jul 03 15:22:49.711275 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.711251 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-m5474"] Jul 03 15:22:49.715300 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:49.715281 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-m5474"] Jul 03 15:22:50.235004 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:22:50.234962 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf1028d-4ab3-49e5-8df5-5f4b602f90d9" path="/var/lib/kubelet/pods/dcf1028d-4ab3-49e5-8df5-5f4b602f90d9/volumes" Jul 03 15:23:17.408445 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:23:17.408330 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:23:17.411288 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:23:17.409785 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:24:17.900088 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.900053 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8"] Jul 03 15:24:17.900525 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.900352 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf1028d-4ab3-49e5-8df5-5f4b602f90d9" containerName="authorino" Jul 03 15:24:17.900525 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.900364 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf1028d-4ab3-49e5-8df5-5f4b602f90d9" containerName="authorino" Jul 03 15:24:17.900525 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.900416 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf1028d-4ab3-49e5-8df5-5f4b602f90d9" containerName="authorino" Jul 03 15:24:17.903093 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.903072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:17.905913 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.905847 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 03 15:24:17.905913 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.905890 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hxtbm\"" Jul 03 15:24:17.907386 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.907368 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 03 15:24:17.907488 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.907475 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 03 15:24:17.915933 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.915521 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8"] Jul 03 15:24:17.983814 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.983775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1094313a-05ef-4888-bdeb-c9a45a66fabb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:17.983814 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.983815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:17.984083 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.983842 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:17.984083 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.983887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:17.984083 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.984009 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkd5d\" (UniqueName: \"kubernetes.io/projected/1094313a-05ef-4888-bdeb-c9a45a66fabb-kube-api-access-fkd5d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:17.984083 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:17.984067 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085191 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085159 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkd5d\" (UniqueName: \"kubernetes.io/projected/1094313a-05ef-4888-bdeb-c9a45a66fabb-kube-api-access-fkd5d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085342 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085212 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085342 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085246 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1094313a-05ef-4888-bdeb-c9a45a66fabb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085342 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085342 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085342 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085639 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085616 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085703 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085679 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.085772 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.085756 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.087534 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.087513 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1094313a-05ef-4888-bdeb-c9a45a66fabb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.087798 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.087778 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1094313a-05ef-4888-bdeb-c9a45a66fabb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.092863 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.092839 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkd5d\" (UniqueName: \"kubernetes.io/projected/1094313a-05ef-4888-bdeb-c9a45a66fabb-kube-api-access-fkd5d\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctf8\" (UID: \"1094313a-05ef-4888-bdeb-c9a45a66fabb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.214216 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.214121 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:18.340894 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.340861 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8"] Jul 03 15:24:18.345956 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:24:18.345927 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1094313a_05ef_4888_bdeb_c9a45a66fabb.slice/crio-a0099306e95daa88f52a78453c540af1d25b33eb599bb59a8d977fc4d2abc292 WatchSource:0}: Error finding container a0099306e95daa88f52a78453c540af1d25b33eb599bb59a8d977fc4d2abc292: Status 404 returned error can't find the container with id a0099306e95daa88f52a78453c540af1d25b33eb599bb59a8d977fc4d2abc292 Jul 03 15:24:18.347194 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:18.347176 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 15:24:19.088714 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:19.088676 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" event={"ID":"1094313a-05ef-4888-bdeb-c9a45a66fabb","Type":"ContainerStarted","Data":"a0099306e95daa88f52a78453c540af1d25b33eb599bb59a8d977fc4d2abc292"} Jul 03 15:24:20.095622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:20.095585 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" event={"ID":"1094313a-05ef-4888-bdeb-c9a45a66fabb","Type":"ContainerStarted","Data":"db82a9a8101f3cb1021bfa022c46a45d1baa373aa5c938e3633b656eb95f1c12"} Jul 03 15:24:20.114834 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:20.114779 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" podStartSLOduration=2.095932547 podStartE2EDuration="3.114762481s" totalImagesPullingTime="1.018829934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:24:20.113376649 +0000 UTC m=+664.345809539" watchObservedRunningTime="2026-07-03 15:24:20.114762481 +0000 UTC m=+664.347195404" Jul 03 15:24:28.214590 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:28.214535 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:28.215155 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:28.214618 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:28.230428 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:28.230404 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:29.146845 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:29.146819 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctf8" Jul 03 15:24:40.201045 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.201005 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq"] Jul 03 15:24:40.207753 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.207729 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.210434 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.210414 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 03 15:24:40.215164 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.215137 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq"] Jul 03 15:24:40.267581 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.267542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.267581 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.267583 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee657f9-a711-4168-a7c1-f66e8522a306-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.267826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.267613 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/0ee657f9-a711-4168-a7c1-f66e8522a306-kube-api-access-qvmcj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.267826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.267693 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.267826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.267746 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.267938 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.267818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.368825 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.368794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369023 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.368832 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee657f9-a711-4168-a7c1-f66e8522a306-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369023 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.368859 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/0ee657f9-a711-4168-a7c1-f66e8522a306-kube-api-access-qvmcj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369023 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.368892 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369023 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.368927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369239 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.369027 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369294 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.369252 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369371 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.369343 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.369407 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.369384 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.371232 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.371201 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee657f9-a711-4168-a7c1-f66e8522a306-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.371325 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.371267 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee657f9-a711-4168-a7c1-f66e8522a306-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.376825 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.376795 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/0ee657f9-a711-4168-a7c1-f66e8522a306-kube-api-access-qvmcj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq\" (UID: \"0ee657f9-a711-4168-a7c1-f66e8522a306\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.518678 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.518588 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:40.652183 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:40.652157 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq"] Jul 03 15:24:40.654823 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:24:40.654796 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee657f9_a711_4168_a7c1_f66e8522a306.slice/crio-6d5eb78d132d17968c0ad04d99bef4a1b79771fe6b2547ac9372ded7ab1ecb0b WatchSource:0}: Error finding container 6d5eb78d132d17968c0ad04d99bef4a1b79771fe6b2547ac9372ded7ab1ecb0b: Status 404 returned error can't find the container with id 6d5eb78d132d17968c0ad04d99bef4a1b79771fe6b2547ac9372ded7ab1ecb0b Jul 03 15:24:41.190633 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:41.190538 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" event={"ID":"0ee657f9-a711-4168-a7c1-f66e8522a306","Type":"ContainerStarted","Data":"8d0808396768a0cb2ee06c4ed4d49710ebde368d8b9090e846e94152300c96fb"} Jul 03 15:24:41.190633 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:41.190584 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" event={"ID":"0ee657f9-a711-4168-a7c1-f66e8522a306","Type":"ContainerStarted","Data":"6d5eb78d132d17968c0ad04d99bef4a1b79771fe6b2547ac9372ded7ab1ecb0b"} Jul 03 15:24:41.209946 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:41.209889 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" podStartSLOduration=1.032691726 podStartE2EDuration="1.209872673s" totalImagesPullingTime="177.180947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:24:41.208016645 +0000 UTC m=+685.440449535" watchObservedRunningTime="2026-07-03 15:24:41.209872673 +0000 UTC m=+685.442305562" Jul 03 15:24:50.519676 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:50.519628 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:50.520112 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:50.519707 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:50.532174 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:50.532146 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:24:51.246000 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:24:51.245967 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq" Jul 03 15:26:11.349843 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.349809 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54b68d65c6-dh6nt"] Jul 03 15:26:11.352883 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.352866 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.355679 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.355659 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 03 15:26:11.355803 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.355764 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 03 15:26:11.355803 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.355779 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q7cf6\"" Jul 03 15:26:11.356950 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.356928 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 03 15:26:11.361766 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.361746 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54b68d65c6-dh6nt"] Jul 03 15:26:11.527153 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.527113 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/938e32af-521a-404e-aac5-0b27a829117e-tls-cert\") pod \"authorino-54b68d65c6-dh6nt\" (UID: \"938e32af-521a-404e-aac5-0b27a829117e\") " pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.527321 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.527186 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdj8\" (UniqueName: \"kubernetes.io/projected/938e32af-521a-404e-aac5-0b27a829117e-kube-api-access-8sdj8\") pod \"authorino-54b68d65c6-dh6nt\" (UID: \"938e32af-521a-404e-aac5-0b27a829117e\") " pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.628462 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.628386 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/938e32af-521a-404e-aac5-0b27a829117e-tls-cert\") pod \"authorino-54b68d65c6-dh6nt\" (UID: \"938e32af-521a-404e-aac5-0b27a829117e\") " pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.628610 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.628469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdj8\" (UniqueName: \"kubernetes.io/projected/938e32af-521a-404e-aac5-0b27a829117e-kube-api-access-8sdj8\") pod \"authorino-54b68d65c6-dh6nt\" (UID: \"938e32af-521a-404e-aac5-0b27a829117e\") " pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.631125 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.631096 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/938e32af-521a-404e-aac5-0b27a829117e-tls-cert\") pod \"authorino-54b68d65c6-dh6nt\" (UID: \"938e32af-521a-404e-aac5-0b27a829117e\") " pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.638732 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.638703 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdj8\" (UniqueName: \"kubernetes.io/projected/938e32af-521a-404e-aac5-0b27a829117e-kube-api-access-8sdj8\") pod \"authorino-54b68d65c6-dh6nt\" (UID: \"938e32af-521a-404e-aac5-0b27a829117e\") " pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.662691 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.662662 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54b68d65c6-dh6nt" Jul 03 15:26:11.802714 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:11.802669 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54b68d65c6-dh6nt"] Jul 03 15:26:11.804863 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:26:11.804838 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod938e32af_521a_404e_aac5_0b27a829117e.slice/crio-98d0207c87326749e6f9ea0e05e891b7be45d4d45b2361f5dc427f6438d58530 WatchSource:0}: Error finding container 98d0207c87326749e6f9ea0e05e891b7be45d4d45b2361f5dc427f6438d58530: Status 404 returned error can't find the container with id 98d0207c87326749e6f9ea0e05e891b7be45d4d45b2361f5dc427f6438d58530 Jul 03 15:26:12.602425 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:12.602383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b68d65c6-dh6nt" event={"ID":"938e32af-521a-404e-aac5-0b27a829117e","Type":"ContainerStarted","Data":"3968d85d0b50a3fa2d7a5a62e1dc30b808bcb359861e84d8becaccc386621bad"} Jul 03 15:26:12.602425 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:12.602433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54b68d65c6-dh6nt" event={"ID":"938e32af-521a-404e-aac5-0b27a829117e","Type":"ContainerStarted","Data":"98d0207c87326749e6f9ea0e05e891b7be45d4d45b2361f5dc427f6438d58530"} Jul 03 15:26:12.617216 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:26:12.617164 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54b68d65c6-dh6nt" podStartSLOduration=1.168819306 podStartE2EDuration="1.617147955s" totalImagesPullingTime="448.328649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:26:12.616278459 +0000 UTC m=+776.848711355" watchObservedRunningTime="2026-07-03 15:26:12.617147955 +0000 UTC m=+776.849580843" Jul 03 15:28:17.436057 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:28:17.435935 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:28:17.441644 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:28:17.438206 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:33:17.463377 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:33:17.463260 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:33:17.467375 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:33:17.465807 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:38:17.492872 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:38:17.492756 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:38:17.496109 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:38:17.494967 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:43:17.524769 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:43:17.524660 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:43:17.530644 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:43:17.526724 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:45:47.311947 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.311906 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm"] Jul 03 15:45:47.315360 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.315339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.319621 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.319594 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-33bce5\"/\"kube-root-ca.crt\"" Jul 03 15:45:47.319750 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.319621 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-33bce5\"/\"default-dockercfg-b7gdl\"" Jul 03 15:45:47.319750 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.319656 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-33bce5\"/\"openshift-service-ca.crt\"" Jul 03 15:45:47.319750 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.319660 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-33bce5\"/\"test-model-33bce5-kserve-self-signed-certs\"" Jul 03 15:45:47.325860 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.325837 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm"] Jul 03 15:45:47.426120 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.426077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-dshm\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.426120 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.426118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-home\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.426381 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.426152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-model-cache\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.426381 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.426208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65ea1d-17c6-4653-871e-10ce67252317-tls-certs\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.426381 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.426304 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dww4\" (UniqueName: \"kubernetes.io/projected/ef65ea1d-17c6-4653-871e-10ce67252317-kube-api-access-2dww4\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.426381 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.426332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-tmp-dir\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.527589 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.527545 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-dshm\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.527589 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.527586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-home\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.527852 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.527617 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-model-cache\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.527852 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.527711 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65ea1d-17c6-4653-871e-10ce67252317-tls-certs\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.527852 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.527767 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dww4\" (UniqueName: \"kubernetes.io/projected/ef65ea1d-17c6-4653-871e-10ce67252317-kube-api-access-2dww4\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.527852 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.527785 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-tmp-dir\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.528147 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.528122 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-home\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.528195 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.528182 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-model-cache\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.528262 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.528222 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-tmp-dir\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.529864 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.529844 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-dshm\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.530147 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.530131 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65ea1d-17c6-4653-871e-10ce67252317-tls-certs\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.545229 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.545202 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dww4\" (UniqueName: \"kubernetes.io/projected/ef65ea1d-17c6-4653-871e-10ce67252317-kube-api-access-2dww4\") pod \"test-model-33bce5-kserve-7c969cf58c-2tpkm\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.626434 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.626377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:47.758323 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.758299 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm"] Jul 03 15:45:47.760327 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:45:47.760290 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef65ea1d_17c6_4653_871e_10ce67252317.slice/crio-b90d66818a6e5c845991caa6d1f2255a4f4d3da361059e7a3c000368073f6bb8 WatchSource:0}: Error finding container b90d66818a6e5c845991caa6d1f2255a4f4d3da361059e7a3c000368073f6bb8: Status 404 returned error can't find the container with id b90d66818a6e5c845991caa6d1f2255a4f4d3da361059e7a3c000368073f6bb8 Jul 03 15:45:47.971432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.971336 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" event={"ID":"ef65ea1d-17c6-4653-871e-10ce67252317","Type":"ContainerStarted","Data":"af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4"} Jul 03 15:45:47.971432 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.971376 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" event={"ID":"ef65ea1d-17c6-4653-871e-10ce67252317","Type":"ContainerStarted","Data":"b90d66818a6e5c845991caa6d1f2255a4f4d3da361059e7a3c000368073f6bb8"} Jul 03 15:45:47.991800 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:47.991739 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" podStartSLOduration=0.991720563 podStartE2EDuration="991.720563ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:45:47.990064231 +0000 UTC m=+1952.222497117" watchObservedRunningTime="2026-07-03 15:45:47.991720563 +0000 UTC m=+1952.224153456" Jul 03 15:45:57.627081 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:57.627044 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:57.627081 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:57.627089 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:57.639252 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:57.639220 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:45:58.030007 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:45:58.029902 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:46:20.899490 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:20.899454 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z"] Jul 03 15:46:20.902512 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:20.902491 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:20.905137 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:20.905114 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-59ce5a\"/\"kube-root-ca.crt\"" Jul 03 15:46:20.905137 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:20.905133 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-59ce5a\"/\"default-dockercfg-vpjqr\"" Jul 03 15:46:20.906399 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:20.906382 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-59ce5a\"/\"test-model-59ce5a-kserve-self-signed-certs\"" Jul 03 15:46:20.906467 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:20.906414 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-59ce5a\"/\"openshift-service-ca.crt\"" Jul 03 15:46:20.912899 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:20.912874 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z"] Jul 03 15:46:21.010710 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.010673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-home\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.010710 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.010710 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-dshm\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.010910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.010732 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzmd\" (UniqueName: \"kubernetes.io/projected/a8c9e862-7fab-4304-a49d-36953a8a22a3-kube-api-access-mmzmd\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.010910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.010791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-tmp-dir\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.010910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.010860 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c9e862-7fab-4304-a49d-36953a8a22a3-tls-certs\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.010910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.010881 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-model-cache\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.111945 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.111905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzmd\" (UniqueName: \"kubernetes.io/projected/a8c9e862-7fab-4304-a49d-36953a8a22a3-kube-api-access-mmzmd\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.111945 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.111949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-tmp-dir\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.112178 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.112100 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c9e862-7fab-4304-a49d-36953a8a22a3-tls-certs\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.112178 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.112140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-model-cache\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.112257 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.112216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-home\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.112257 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.112240 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-dshm\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.112354 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.112336 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-tmp-dir\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.112575 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.112552 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-model-cache\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.112658 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.112560 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-home\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.114364 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.114339 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-dshm\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.114528 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.114510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c9e862-7fab-4304-a49d-36953a8a22a3-tls-certs\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.121223 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.121200 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzmd\" (UniqueName: \"kubernetes.io/projected/a8c9e862-7fab-4304-a49d-36953a8a22a3-kube-api-access-mmzmd\") pod \"test-model-59ce5a-kserve-74d9778fd9-cw29z\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.212404 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.212309 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:21.346909 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:21.346874 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z"] Jul 03 15:46:21.349089 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:46:21.349055 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8c9e862_7fab_4304_a49d_36953a8a22a3.slice/crio-c3d2bd9277cead15d43f829e0e040e45767bd6e36b4198e03cccdd865f5a6946 WatchSource:0}: Error finding container c3d2bd9277cead15d43f829e0e040e45767bd6e36b4198e03cccdd865f5a6946: Status 404 returned error can't find the container with id c3d2bd9277cead15d43f829e0e040e45767bd6e36b4198e03cccdd865f5a6946 Jul 03 15:46:22.130067 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:22.130028 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" event={"ID":"a8c9e862-7fab-4304-a49d-36953a8a22a3","Type":"ContainerStarted","Data":"1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141"} Jul 03 15:46:22.130423 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:22.130077 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" event={"ID":"a8c9e862-7fab-4304-a49d-36953a8a22a3","Type":"ContainerStarted","Data":"c3d2bd9277cead15d43f829e0e040e45767bd6e36b4198e03cccdd865f5a6946"} Jul 03 15:46:22.150492 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:22.150443 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" podStartSLOduration=2.150427862 podStartE2EDuration="2.150427862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:46:22.149235804 +0000 UTC m=+1986.381668695" watchObservedRunningTime="2026-07-03 15:46:22.150427862 +0000 UTC m=+1986.382860750" Jul 03 15:46:31.213136 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:31.213101 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:31.213591 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:31.213152 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:31.225656 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:31.225629 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:32.188755 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:32.188725 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:40.809371 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:40.809333 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm"] Jul 03 15:46:41.789938 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:41.789903 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z"] Jul 03 15:46:55.828915 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:55.828851 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" podUID="ef65ea1d-17c6-4653-871e-10ce67252317" containerName="main" containerID="cri-o://af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4" gracePeriod=45 Jul 03 15:46:56.069937 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.069909 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:46:56.121437 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121405 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-dshm\") pod \"ef65ea1d-17c6-4653-871e-10ce67252317\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " Jul 03 15:46:56.121622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121449 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dww4\" (UniqueName: \"kubernetes.io/projected/ef65ea1d-17c6-4653-871e-10ce67252317-kube-api-access-2dww4\") pod \"ef65ea1d-17c6-4653-871e-10ce67252317\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " Jul 03 15:46:56.121622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121470 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-tmp-dir\") pod \"ef65ea1d-17c6-4653-871e-10ce67252317\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " Jul 03 15:46:56.121622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121489 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65ea1d-17c6-4653-871e-10ce67252317-tls-certs\") pod \"ef65ea1d-17c6-4653-871e-10ce67252317\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " Jul 03 15:46:56.121622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121516 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-home\") pod \"ef65ea1d-17c6-4653-871e-10ce67252317\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " Jul 03 15:46:56.121622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121558 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-model-cache\") pod \"ef65ea1d-17c6-4653-871e-10ce67252317\" (UID: \"ef65ea1d-17c6-4653-871e-10ce67252317\") " Jul 03 15:46:56.121883 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121734 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef65ea1d-17c6-4653-871e-10ce67252317" (UID: "ef65ea1d-17c6-4653-871e-10ce67252317"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:56.121939 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121846 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-home" (OuterVolumeSpecName: "home") pod "ef65ea1d-17c6-4653-871e-10ce67252317" (UID: "ef65ea1d-17c6-4653-871e-10ce67252317"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:56.121939 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.121912 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef65ea1d-17c6-4653-871e-10ce67252317" (UID: "ef65ea1d-17c6-4653-871e-10ce67252317"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:56.123717 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.123695 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-dshm" (OuterVolumeSpecName: "dshm") pod "ef65ea1d-17c6-4653-871e-10ce67252317" (UID: "ef65ea1d-17c6-4653-871e-10ce67252317"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:56.123838 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.123816 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef65ea1d-17c6-4653-871e-10ce67252317-kube-api-access-2dww4" (OuterVolumeSpecName: "kube-api-access-2dww4") pod "ef65ea1d-17c6-4653-871e-10ce67252317" (UID: "ef65ea1d-17c6-4653-871e-10ce67252317"). InnerVolumeSpecName "kube-api-access-2dww4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:46:56.123838 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.123824 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef65ea1d-17c6-4653-871e-10ce67252317-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef65ea1d-17c6-4653-871e-10ce67252317" (UID: "ef65ea1d-17c6-4653-871e-10ce67252317"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:46:56.222184 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.222144 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-model-cache\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:56.222184 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.222176 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-dshm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:56.222184 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.222185 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dww4\" (UniqueName: \"kubernetes.io/projected/ef65ea1d-17c6-4653-871e-10ce67252317-kube-api-access-2dww4\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:56.222184 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.222195 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-tmp-dir\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:56.222433 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.222204 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef65ea1d-17c6-4653-871e-10ce67252317-tls-certs\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:56.222433 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.222212 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef65ea1d-17c6-4653-871e-10ce67252317-home\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:56.286431 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.286399 2572 generic.go:356] "Generic (PLEG): container finished" podID="ef65ea1d-17c6-4653-871e-10ce67252317" containerID="af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4" exitCode=0 Jul 03 15:46:56.286431 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.286435 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" event={"ID":"ef65ea1d-17c6-4653-871e-10ce67252317","Type":"ContainerDied","Data":"af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4"} Jul 03 15:46:56.286642 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.286448 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" Jul 03 15:46:56.286642 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.286462 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm" event={"ID":"ef65ea1d-17c6-4653-871e-10ce67252317","Type":"ContainerDied","Data":"b90d66818a6e5c845991caa6d1f2255a4f4d3da361059e7a3c000368073f6bb8"} Jul 03 15:46:56.286642 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.286480 2572 scope.go:117] "RemoveContainer" containerID="af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4" Jul 03 15:46:56.295104 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.295070 2572 scope.go:117] "RemoveContainer" containerID="af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4" Jul 03 15:46:56.295340 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:46:56.295319 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4\": container with ID starting with af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4 not found: ID does not exist" containerID="af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4" Jul 03 15:46:56.295383 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.295348 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4"} err="failed to get container status \"af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4\": rpc error: code = NotFound desc = could not find container \"af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4\": container with ID starting with af98bbb22d5a7a55f831abd5a2d3911e6b269ec4eeeba021c541922a95461dd4 not found: ID does not exist" Jul 03 15:46:56.303758 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.303735 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm"] Jul 03 15:46:56.308100 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.308077 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-33bce5/test-model-33bce5-kserve-7c969cf58c-2tpkm"] Jul 03 15:46:56.810521 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:56.810456 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" podUID="a8c9e862-7fab-4304-a49d-36953a8a22a3" containerName="main" containerID="cri-o://1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141" gracePeriod=45 Jul 03 15:46:57.059840 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.059815 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:57.129157 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129123 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-tmp-dir\") pod \"a8c9e862-7fab-4304-a49d-36953a8a22a3\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " Jul 03 15:46:57.129314 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129189 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c9e862-7fab-4304-a49d-36953a8a22a3-tls-certs\") pod \"a8c9e862-7fab-4304-a49d-36953a8a22a3\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " Jul 03 15:46:57.129314 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129233 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-home\") pod \"a8c9e862-7fab-4304-a49d-36953a8a22a3\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " Jul 03 15:46:57.129314 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129279 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmzmd\" (UniqueName: \"kubernetes.io/projected/a8c9e862-7fab-4304-a49d-36953a8a22a3-kube-api-access-mmzmd\") pod \"a8c9e862-7fab-4304-a49d-36953a8a22a3\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " Jul 03 15:46:57.129314 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129304 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-dshm\") pod \"a8c9e862-7fab-4304-a49d-36953a8a22a3\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " Jul 03 15:46:57.129543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129449 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8c9e862-7fab-4304-a49d-36953a8a22a3" (UID: "a8c9e862-7fab-4304-a49d-36953a8a22a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:57.129543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129530 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-tmp-dir\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:57.129643 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.129579 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-home" (OuterVolumeSpecName: "home") pod "a8c9e862-7fab-4304-a49d-36953a8a22a3" (UID: "a8c9e862-7fab-4304-a49d-36953a8a22a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:57.131472 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.131445 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-dshm" (OuterVolumeSpecName: "dshm") pod "a8c9e862-7fab-4304-a49d-36953a8a22a3" (UID: "a8c9e862-7fab-4304-a49d-36953a8a22a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:57.131580 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.131471 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c9e862-7fab-4304-a49d-36953a8a22a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8c9e862-7fab-4304-a49d-36953a8a22a3" (UID: "a8c9e862-7fab-4304-a49d-36953a8a22a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:46:57.131580 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.131558 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c9e862-7fab-4304-a49d-36953a8a22a3-kube-api-access-mmzmd" (OuterVolumeSpecName: "kube-api-access-mmzmd") pod "a8c9e862-7fab-4304-a49d-36953a8a22a3" (UID: "a8c9e862-7fab-4304-a49d-36953a8a22a3"). InnerVolumeSpecName "kube-api-access-mmzmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:46:57.230278 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.230241 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-model-cache\") pod \"a8c9e862-7fab-4304-a49d-36953a8a22a3\" (UID: \"a8c9e862-7fab-4304-a49d-36953a8a22a3\") " Jul 03 15:46:57.230489 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.230462 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8c9e862-7fab-4304-a49d-36953a8a22a3" (UID: "a8c9e862-7fab-4304-a49d-36953a8a22a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:46:57.230570 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.230520 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-home\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:57.230570 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.230541 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmzmd\" (UniqueName: \"kubernetes.io/projected/a8c9e862-7fab-4304-a49d-36953a8a22a3-kube-api-access-mmzmd\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:57.230570 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.230557 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-dshm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:57.230674 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.230570 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c9e862-7fab-4304-a49d-36953a8a22a3-tls-certs\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:57.292906 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.292859 2572 generic.go:356] "Generic (PLEG): container finished" podID="a8c9e862-7fab-4304-a49d-36953a8a22a3" containerID="1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141" exitCode=0 Jul 03 15:46:57.292906 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.292909 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" Jul 03 15:46:57.293177 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.292946 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" event={"ID":"a8c9e862-7fab-4304-a49d-36953a8a22a3","Type":"ContainerDied","Data":"1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141"} Jul 03 15:46:57.293177 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.293012 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z" event={"ID":"a8c9e862-7fab-4304-a49d-36953a8a22a3","Type":"ContainerDied","Data":"c3d2bd9277cead15d43f829e0e040e45767bd6e36b4198e03cccdd865f5a6946"} Jul 03 15:46:57.293177 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.293037 2572 scope.go:117] "RemoveContainer" containerID="1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141" Jul 03 15:46:57.302064 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.302044 2572 scope.go:117] "RemoveContainer" containerID="1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141" Jul 03 15:46:57.302336 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:46:57.302318 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141\": container with ID starting with 1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141 not found: ID does not exist" containerID="1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141" Jul 03 15:46:57.302390 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.302345 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141"} err="failed to get container status \"1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141\": rpc error: code = NotFound desc = could not find container \"1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141\": container with ID starting with 1d71c8223f1423642ba4169189da5319c27c6c0a435bce5f0dbfd55007896141 not found: ID does not exist" Jul 03 15:46:57.314615 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.314590 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z"] Jul 03 15:46:57.316724 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.316704 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-59ce5a/test-model-59ce5a-kserve-74d9778fd9-cw29z"] Jul 03 15:46:57.331005 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:57.330959 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e862-7fab-4304-a49d-36953a8a22a3-model-cache\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:46:58.231354 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:58.231325 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c9e862-7fab-4304-a49d-36953a8a22a3" path="/var/lib/kubelet/pods/a8c9e862-7fab-4304-a49d-36953a8a22a3/volumes" Jul 03 15:46:58.231725 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:46:58.231699 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef65ea1d-17c6-4653-871e-10ce67252317" path="/var/lib/kubelet/pods/ef65ea1d-17c6-4653-871e-10ce67252317/volumes" Jul 03 15:48:17.553319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:17.553195 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:48:17.556538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:17.555971 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:48:38.107682 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.107649 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd"] Jul 03 15:48:38.108161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.107941 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c9e862-7fab-4304-a49d-36953a8a22a3" containerName="main" Jul 03 15:48:38.108161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.107951 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c9e862-7fab-4304-a49d-36953a8a22a3" containerName="main" Jul 03 15:48:38.108161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.107964 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef65ea1d-17c6-4653-871e-10ce67252317" containerName="main" Jul 03 15:48:38.108161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.107969 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef65ea1d-17c6-4653-871e-10ce67252317" containerName="main" Jul 03 15:48:38.108161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.108029 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef65ea1d-17c6-4653-871e-10ce67252317" containerName="main" Jul 03 15:48:38.108161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.108037 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c9e862-7fab-4304-a49d-36953a8a22a3" containerName="main" Jul 03 15:48:38.110758 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.110741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.115520 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.115495 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"auth-test-model-3703d4-kserve-self-signed-certs\"" Jul 03 15:48:38.115661 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.115500 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"default-dockercfg-tcx7m\"" Jul 03 15:48:38.115661 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.115551 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"kube-root-ca.crt\"" Jul 03 15:48:38.115661 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.115586 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"openshift-service-ca.crt\"" Jul 03 15:48:38.127501 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.127472 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd"] Jul 03 15:48:38.274241 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.274205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-home\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.274241 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.274240 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tmp-dir\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.274465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.274281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tls-certs\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.274465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.274355 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhqk\" (UniqueName: \"kubernetes.io/projected/4ea42010-1c51-4a23-8738-8fbc73fd95e4-kube-api-access-9hhqk\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.274465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.274414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-dshm\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.274465 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.274446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-model-cache\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.375652 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.375566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-home\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.375652 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.375603 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tmp-dir\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.375879 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.375663 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tls-certs\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.376084 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.376061 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhqk\" (UniqueName: \"kubernetes.io/projected/4ea42010-1c51-4a23-8738-8fbc73fd95e4-kube-api-access-9hhqk\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.376184 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.376086 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tmp-dir\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.376184 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.376112 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-home\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.376184 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.376162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-dshm\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.376336 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.376215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-model-cache\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.376488 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.376471 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-model-cache\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.378169 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.378147 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-dshm\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.378275 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.378261 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tls-certs\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.385106 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.385079 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhqk\" (UniqueName: \"kubernetes.io/projected/4ea42010-1c51-4a23-8738-8fbc73fd95e4-kube-api-access-9hhqk\") pod \"auth-test-model-3703d4-kserve-5859fff847-hvlhd\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.420063 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.420031 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:38.550826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.550803 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd"] Jul 03 15:48:38.553255 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:48:38.553223 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea42010_1c51_4a23_8738_8fbc73fd95e4.slice/crio-2d5f998af706dfbf6a0c859ac864d6d8c72186fb0e56f41a703bccd01488d6f1 WatchSource:0}: Error finding container 2d5f998af706dfbf6a0c859ac864d6d8c72186fb0e56f41a703bccd01488d6f1: Status 404 returned error can't find the container with id 2d5f998af706dfbf6a0c859ac864d6d8c72186fb0e56f41a703bccd01488d6f1 Jul 03 15:48:38.753666 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.753571 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" event={"ID":"4ea42010-1c51-4a23-8738-8fbc73fd95e4","Type":"ContainerStarted","Data":"b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5"} Jul 03 15:48:38.753666 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.753617 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" event={"ID":"4ea42010-1c51-4a23-8738-8fbc73fd95e4","Type":"ContainerStarted","Data":"2d5f998af706dfbf6a0c859ac864d6d8c72186fb0e56f41a703bccd01488d6f1"} Jul 03 15:48:38.773435 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:38.773379 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" podStartSLOduration=0.773365638 podStartE2EDuration="773.365638ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:48:38.771741309 +0000 UTC m=+2123.004174195" watchObservedRunningTime="2026-07-03 15:48:38.773365638 +0000 UTC m=+2123.005798530" Jul 03 15:48:48.421110 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:48.421068 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:48.421567 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:48.421129 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:48.433617 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:48.433592 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:48:48.809463 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:48:48.809383 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:49:17.009693 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.009657 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv"] Jul 03 15:49:17.012956 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.012941 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.016328 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.016306 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"kube-root-ca.crt\"" Jul 03 15:49:17.017584 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.017564 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"default-dockercfg-bz8cf\"" Jul 03 15:49:17.017649 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.017613 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"openshift-service-ca.crt\"" Jul 03 15:49:17.017700 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.017611 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"auth-test-model-31658c-kserve-self-signed-certs\"" Jul 03 15:49:17.025582 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.025561 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv"] Jul 03 15:49:17.102292 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.102261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-tmp-dir\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.102292 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.102297 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbhq\" (UniqueName: \"kubernetes.io/projected/c7011301-ac53-4bdc-995d-e4388704226f-kube-api-access-7nbhq\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.102554 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.102317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-dshm\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.102554 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.102373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-home\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.102554 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.102416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-model-cache\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.102554 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.102537 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7011301-ac53-4bdc-995d-e4388704226f-tls-certs\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.203745 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.203704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbhq\" (UniqueName: \"kubernetes.io/projected/c7011301-ac53-4bdc-995d-e4388704226f-kube-api-access-7nbhq\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.203745 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.203748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-dshm\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.204044 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.203768 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-home\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.204044 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.203886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-model-cache\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.204164 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.204046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7011301-ac53-4bdc-995d-e4388704226f-tls-certs\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.204164 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.204102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-tmp-dir\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.204272 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.204191 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-home\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.204326 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.204283 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-model-cache\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.204409 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.204388 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-tmp-dir\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.206079 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.206055 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-dshm\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.206518 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.206495 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7011301-ac53-4bdc-995d-e4388704226f-tls-certs\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.212159 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.212140 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbhq\" (UniqueName: \"kubernetes.io/projected/c7011301-ac53-4bdc-995d-e4388704226f-kube-api-access-7nbhq\") pod \"auth-test-model-31658c-kserve-766c457c75-l84bv\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.321819 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.321733 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:17.457857 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.457830 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv"] Jul 03 15:49:17.459745 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:49:17.459717 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7011301_ac53_4bdc_995d_e4388704226f.slice/crio-3ed3dcc7f2a4d35bef78ea705cf6ac145632178a9ec1884b602d6f7851e82701 WatchSource:0}: Error finding container 3ed3dcc7f2a4d35bef78ea705cf6ac145632178a9ec1884b602d6f7851e82701: Status 404 returned error can't find the container with id 3ed3dcc7f2a4d35bef78ea705cf6ac145632178a9ec1884b602d6f7851e82701 Jul 03 15:49:17.936785 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.936751 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" event={"ID":"c7011301-ac53-4bdc-995d-e4388704226f","Type":"ContainerStarted","Data":"78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08"} Jul 03 15:49:17.936785 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.936790 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" event={"ID":"c7011301-ac53-4bdc-995d-e4388704226f","Type":"ContainerStarted","Data":"3ed3dcc7f2a4d35bef78ea705cf6ac145632178a9ec1884b602d6f7851e82701"} Jul 03 15:49:17.956072 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:17.956025 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" podStartSLOduration=1.956009696 podStartE2EDuration="1.956009696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:49:17.955285381 +0000 UTC m=+2162.187718280" watchObservedRunningTime="2026-07-03 15:49:17.956009696 +0000 UTC m=+2162.188442576" Jul 03 15:49:27.322161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:27.322120 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:27.322660 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:27.322183 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:27.334785 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:27.334752 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:49:27.993178 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:49:27.993146 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:51:17.117723 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:17.117686 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd"] Jul 03 15:51:18.078690 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:18.078659 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv"] Jul 03 15:51:32.137224 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.137161 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" podUID="4ea42010-1c51-4a23-8738-8fbc73fd95e4" containerName="main" containerID="cri-o://b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5" gracePeriod=45 Jul 03 15:51:32.391742 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.391682 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:51:32.567004 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.566952 2572 generic.go:356] "Generic (PLEG): container finished" podID="4ea42010-1c51-4a23-8738-8fbc73fd95e4" containerID="b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5" exitCode=0 Jul 03 15:51:32.567166 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567016 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" event={"ID":"4ea42010-1c51-4a23-8738-8fbc73fd95e4","Type":"ContainerDied","Data":"b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5"} Jul 03 15:51:32.567166 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567019 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" Jul 03 15:51:32.567166 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567046 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd" event={"ID":"4ea42010-1c51-4a23-8738-8fbc73fd95e4","Type":"ContainerDied","Data":"2d5f998af706dfbf6a0c859ac864d6d8c72186fb0e56f41a703bccd01488d6f1"} Jul 03 15:51:32.567166 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567062 2572 scope.go:117] "RemoveContainer" containerID="b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5" Jul 03 15:51:32.567519 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567506 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-model-cache\") pod \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " Jul 03 15:51:32.567594 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567578 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tmp-dir\") pod \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " Jul 03 15:51:32.567634 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567613 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-home\") pod \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " Jul 03 15:51:32.567669 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567652 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-dshm\") pod \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " Jul 03 15:51:32.567705 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567690 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tls-certs\") pod \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " Jul 03 15:51:32.567767 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567751 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hhqk\" (UniqueName: \"kubernetes.io/projected/4ea42010-1c51-4a23-8738-8fbc73fd95e4-kube-api-access-9hhqk\") pod \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\" (UID: \"4ea42010-1c51-4a23-8738-8fbc73fd95e4\") " Jul 03 15:51:32.567826 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567796 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ea42010-1c51-4a23-8738-8fbc73fd95e4" (UID: "4ea42010-1c51-4a23-8738-8fbc73fd95e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:32.568050 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.567916 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ea42010-1c51-4a23-8738-8fbc73fd95e4" (UID: "4ea42010-1c51-4a23-8738-8fbc73fd95e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:32.568050 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.568004 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-home" (OuterVolumeSpecName: "home") pod "4ea42010-1c51-4a23-8738-8fbc73fd95e4" (UID: "4ea42010-1c51-4a23-8738-8fbc73fd95e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:32.568215 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.568066 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-model-cache\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:32.568215 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.568082 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tmp-dir\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:32.569961 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.569938 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-dshm" (OuterVolumeSpecName: "dshm") pod "4ea42010-1c51-4a23-8738-8fbc73fd95e4" (UID: "4ea42010-1c51-4a23-8738-8fbc73fd95e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:32.570242 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.570216 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea42010-1c51-4a23-8738-8fbc73fd95e4-kube-api-access-9hhqk" (OuterVolumeSpecName: "kube-api-access-9hhqk") pod "4ea42010-1c51-4a23-8738-8fbc73fd95e4" (UID: "4ea42010-1c51-4a23-8738-8fbc73fd95e4"). InnerVolumeSpecName "kube-api-access-9hhqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:51:32.570242 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.570218 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ea42010-1c51-4a23-8738-8fbc73fd95e4" (UID: "4ea42010-1c51-4a23-8738-8fbc73fd95e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:51:32.585130 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.585103 2572 scope.go:117] "RemoveContainer" containerID="b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5" Jul 03 15:51:32.585427 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:51:32.585403 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5\": container with ID starting with b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5 not found: ID does not exist" containerID="b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5" Jul 03 15:51:32.585493 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.585437 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5"} err="failed to get container status \"b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5\": rpc error: code = NotFound desc = could not find container \"b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5\": container with ID starting with b675406e193000c7f27c61da115c1ec46fa74154386f907aebe19cf5982d55f5 not found: ID does not exist" Jul 03 15:51:32.669466 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.669376 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-home\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:32.669466 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.669420 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea42010-1c51-4a23-8738-8fbc73fd95e4-dshm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:32.669466 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.669428 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea42010-1c51-4a23-8738-8fbc73fd95e4-tls-certs\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:32.669466 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.669437 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhqk\" (UniqueName: \"kubernetes.io/projected/4ea42010-1c51-4a23-8738-8fbc73fd95e4-kube-api-access-9hhqk\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:32.888103 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.888074 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd"] Jul 03 15:51:32.896458 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:32.891936 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/auth-test-model-3703d4-kserve-5859fff847-hvlhd"] Jul 03 15:51:33.098788 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.098730 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" podUID="c7011301-ac53-4bdc-995d-e4388704226f" containerName="main" containerID="cri-o://78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08" gracePeriod=45 Jul 03 15:51:33.349684 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.349620 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:51:33.375346 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375312 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nbhq\" (UniqueName: \"kubernetes.io/projected/c7011301-ac53-4bdc-995d-e4388704226f-kube-api-access-7nbhq\") pod \"c7011301-ac53-4bdc-995d-e4388704226f\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " Jul 03 15:51:33.375346 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375351 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-model-cache\") pod \"c7011301-ac53-4bdc-995d-e4388704226f\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " Jul 03 15:51:33.375582 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375376 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-home\") pod \"c7011301-ac53-4bdc-995d-e4388704226f\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " Jul 03 15:51:33.375582 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375400 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7011301-ac53-4bdc-995d-e4388704226f-tls-certs\") pod \"c7011301-ac53-4bdc-995d-e4388704226f\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " Jul 03 15:51:33.375582 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375478 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-tmp-dir\") pod \"c7011301-ac53-4bdc-995d-e4388704226f\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " Jul 03 15:51:33.375582 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375511 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-dshm\") pod \"c7011301-ac53-4bdc-995d-e4388704226f\" (UID: \"c7011301-ac53-4bdc-995d-e4388704226f\") " Jul 03 15:51:33.375770 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375667 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7011301-ac53-4bdc-995d-e4388704226f" (UID: "c7011301-ac53-4bdc-995d-e4388704226f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:33.375770 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375742 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-model-cache\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:33.375882 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375859 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-home" (OuterVolumeSpecName: "home") pod "c7011301-ac53-4bdc-995d-e4388704226f" (UID: "c7011301-ac53-4bdc-995d-e4388704226f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:33.375929 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.375875 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7011301-ac53-4bdc-995d-e4388704226f" (UID: "c7011301-ac53-4bdc-995d-e4388704226f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:33.377754 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.377724 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7011301-ac53-4bdc-995d-e4388704226f-kube-api-access-7nbhq" (OuterVolumeSpecName: "kube-api-access-7nbhq") pod "c7011301-ac53-4bdc-995d-e4388704226f" (UID: "c7011301-ac53-4bdc-995d-e4388704226f"). InnerVolumeSpecName "kube-api-access-7nbhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:51:33.378145 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.378120 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-dshm" (OuterVolumeSpecName: "dshm") pod "c7011301-ac53-4bdc-995d-e4388704226f" (UID: "c7011301-ac53-4bdc-995d-e4388704226f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:51:33.378249 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.378125 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7011301-ac53-4bdc-995d-e4388704226f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7011301-ac53-4bdc-995d-e4388704226f" (UID: "c7011301-ac53-4bdc-995d-e4388704226f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:51:33.476743 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.476700 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nbhq\" (UniqueName: \"kubernetes.io/projected/c7011301-ac53-4bdc-995d-e4388704226f-kube-api-access-7nbhq\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:33.476743 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.476734 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-home\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:33.476743 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.476745 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7011301-ac53-4bdc-995d-e4388704226f-tls-certs\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:33.477015 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.476776 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-tmp-dir\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:33.477015 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.476785 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7011301-ac53-4bdc-995d-e4388704226f-dshm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:51:33.573958 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.573920 2572 generic.go:356] "Generic (PLEG): container finished" podID="c7011301-ac53-4bdc-995d-e4388704226f" containerID="78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08" exitCode=0 Jul 03 15:51:33.574159 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.573972 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" Jul 03 15:51:33.574159 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.574027 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" event={"ID":"c7011301-ac53-4bdc-995d-e4388704226f","Type":"ContainerDied","Data":"78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08"} Jul 03 15:51:33.574159 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.574081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv" event={"ID":"c7011301-ac53-4bdc-995d-e4388704226f","Type":"ContainerDied","Data":"3ed3dcc7f2a4d35bef78ea705cf6ac145632178a9ec1884b602d6f7851e82701"} Jul 03 15:51:33.574159 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.574104 2572 scope.go:117] "RemoveContainer" containerID="78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08" Jul 03 15:51:33.582892 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.582876 2572 scope.go:117] "RemoveContainer" containerID="78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08" Jul 03 15:51:33.583176 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:51:33.583162 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08\": container with ID starting with 78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08 not found: ID does not exist" containerID="78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08" Jul 03 15:51:33.583222 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.583184 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08"} err="failed to get container status \"78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08\": rpc error: code = NotFound desc = could not find container \"78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08\": container with ID starting with 78c0a2ea00f5e303a8049267968041b763ceb388f40b72de1fdacfa4f025ee08 not found: ID does not exist" Jul 03 15:51:33.594589 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.594563 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv"] Jul 03 15:51:33.596935 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:33.596912 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/auth-test-model-31658c-kserve-766c457c75-l84bv"] Jul 03 15:51:34.231626 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:34.231593 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea42010-1c51-4a23-8738-8fbc73fd95e4" path="/var/lib/kubelet/pods/4ea42010-1c51-4a23-8738-8fbc73fd95e4/volumes" Jul 03 15:51:34.231977 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:34.231964 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7011301-ac53-4bdc-995d-e4388704226f" path="/var/lib/kubelet/pods/c7011301-ac53-4bdc-995d-e4388704226f/volumes" Jul 03 15:51:52.496426 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.496386 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z"] Jul 03 15:51:52.496910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.496677 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea42010-1c51-4a23-8738-8fbc73fd95e4" containerName="main" Jul 03 15:51:52.496910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.496688 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea42010-1c51-4a23-8738-8fbc73fd95e4" containerName="main" Jul 03 15:51:52.496910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.496699 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7011301-ac53-4bdc-995d-e4388704226f" containerName="main" Jul 03 15:51:52.496910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.496704 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7011301-ac53-4bdc-995d-e4388704226f" containerName="main" Jul 03 15:51:52.496910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.496757 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea42010-1c51-4a23-8738-8fbc73fd95e4" containerName="main" Jul 03 15:51:52.496910 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.496765 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7011301-ac53-4bdc-995d-e4388704226f" containerName="main" Jul 03 15:51:52.499503 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.499487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.503390 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.503327 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"default-dockercfg-bz8cf\"" Jul 03 15:51:52.503569 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.503389 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"sub-test-model-31658c-kserve-self-signed-certs\"" Jul 03 15:51:52.503569 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.503490 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"kube-root-ca.crt\"" Jul 03 15:51:52.503569 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.503534 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"openshift-service-ca.crt\"" Jul 03 15:51:52.508911 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.508890 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z"] Jul 03 15:51:52.532673 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.532627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwjp\" (UniqueName: \"kubernetes.io/projected/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-kube-api-access-6vwjp\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.532855 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.532692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-home\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.532855 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.532737 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tmp-dir\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.532855 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.532770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tls-certs\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.532855 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.532810 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-dshm\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.533036 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.532884 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-model-cache\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.633847 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.633813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-home\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.633847 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.633852 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tmp-dir\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.634155 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.633877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tls-certs\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.634155 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.634030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-dshm\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.634155 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.634104 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-model-cache\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.634308 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.634152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwjp\" (UniqueName: \"kubernetes.io/projected/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-kube-api-access-6vwjp\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.634359 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.634326 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tmp-dir\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.634359 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.634342 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-home\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.634551 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.634529 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-model-cache\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.636290 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.636274 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-dshm\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.636463 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.636449 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tls-certs\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.642709 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.642687 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwjp\" (UniqueName: \"kubernetes.io/projected/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-kube-api-access-6vwjp\") pod \"sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.809487 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.809401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:51:52.940762 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:52.940736 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z"] Jul 03 15:51:53.674463 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:53.674415 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" event={"ID":"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76","Type":"ContainerStarted","Data":"9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c"} Jul 03 15:51:53.674463 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:53.674474 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" event={"ID":"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76","Type":"ContainerStarted","Data":"006ebcc0baefb7fd601459479545b61b31dd93670da3c360b8e21a8bb690de06"} Jul 03 15:51:53.693324 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:51:53.693265 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" podStartSLOduration=1.6932484140000001 podStartE2EDuration="1.693248414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:51:53.690339082 +0000 UTC m=+2317.922771971" watchObservedRunningTime="2026-07-03 15:51:53.693248414 +0000 UTC m=+2317.925681303" Jul 03 15:52:02.810216 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:02.810171 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:52:02.810706 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:02.810238 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:52:02.822579 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:02.822557 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:52:03.736684 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:03.736655 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:52:35.497825 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:35.497788 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z"] Jul 03 15:52:45.503565 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.503527 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz"] Jul 03 15:52:45.507509 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.507482 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.510078 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.510052 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"kube-root-ca.crt\"" Jul 03 15:52:45.511568 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.511546 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"openshift-service-ca.crt\"" Jul 03 15:52:45.511773 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.511747 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"rate-test-model-3703d4-kserve-self-signed-certs\"" Jul 03 15:52:45.511845 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.511804 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3703d4\"/\"default-dockercfg-tcx7m\"" Jul 03 15:52:45.514276 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.514256 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz"] Jul 03 15:52:45.583800 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.583764 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frrc\" (UniqueName: \"kubernetes.io/projected/64ca9f41-9940-42e0-a284-8d4aad3784b5-kube-api-access-7frrc\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.583800 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.583804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-model-cache\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.584042 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.583836 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ca9f41-9940-42e0-a284-8d4aad3784b5-tls-certs\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.584042 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.583856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-home\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.584042 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.583951 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-tmp-dir\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.584140 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.584049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-dshm\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685127 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-model-cache\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685127 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ca9f41-9940-42e0-a284-8d4aad3784b5-tls-certs\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685415 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-home\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685415 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-tmp-dir\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685415 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685248 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-dshm\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685415 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685287 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frrc\" (UniqueName: \"kubernetes.io/projected/64ca9f41-9940-42e0-a284-8d4aad3784b5-kube-api-access-7frrc\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685623 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685583 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-home\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685680 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685643 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-model-cache\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.685744 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.685689 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-tmp-dir\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.687449 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.687427 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-dshm\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.687779 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.687759 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ca9f41-9940-42e0-a284-8d4aad3784b5-tls-certs\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.693188 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.693162 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frrc\" (UniqueName: \"kubernetes.io/projected/64ca9f41-9940-42e0-a284-8d4aad3784b5-kube-api-access-7frrc\") pod \"rate-test-model-3703d4-kserve-845687f9ff-ghjsz\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.818634 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.818542 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:45.948915 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:45.948889 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz"] Jul 03 15:52:45.950790 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:52:45.950765 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64ca9f41_9940_42e0_a284_8d4aad3784b5.slice/crio-68c22c226efec0bb5f0f039469b4fdb4d5c8fefdf54a84c90b57ca5450573931 WatchSource:0}: Error finding container 68c22c226efec0bb5f0f039469b4fdb4d5c8fefdf54a84c90b57ca5450573931: Status 404 returned error can't find the container with id 68c22c226efec0bb5f0f039469b4fdb4d5c8fefdf54a84c90b57ca5450573931 Jul 03 15:52:46.919007 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:46.918957 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" event={"ID":"64ca9f41-9940-42e0-a284-8d4aad3784b5","Type":"ContainerStarted","Data":"cb0b9dc42311dfdd7c7101707947064af1dfbb96953c998cfc3774407fa3fd0a"} Jul 03 15:52:46.919543 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:46.919019 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" event={"ID":"64ca9f41-9940-42e0-a284-8d4aad3784b5","Type":"ContainerStarted","Data":"68c22c226efec0bb5f0f039469b4fdb4d5c8fefdf54a84c90b57ca5450573931"} Jul 03 15:52:46.935611 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:46.935564 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" podStartSLOduration=1.9355487139999998 podStartE2EDuration="1.935548714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:52:46.934585039 +0000 UTC m=+2371.167017929" watchObservedRunningTime="2026-07-03 15:52:46.935548714 +0000 UTC m=+2371.167981600" Jul 03 15:52:50.520255 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.520216 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" podUID="817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" containerName="main" containerID="cri-o://9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c" gracePeriod=45 Jul 03 15:52:50.768217 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.768192 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:52:50.830054 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.829937 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tls-certs\") pod \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " Jul 03 15:52:50.830054 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830044 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-home\") pod \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " Jul 03 15:52:50.830248 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830074 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-dshm\") pod \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " Jul 03 15:52:50.830248 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830097 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-model-cache\") pod \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " Jul 03 15:52:50.830248 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830157 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vwjp\" (UniqueName: \"kubernetes.io/projected/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-kube-api-access-6vwjp\") pod \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " Jul 03 15:52:50.830248 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830212 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tmp-dir\") pod \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\" (UID: \"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76\") " Jul 03 15:52:50.830453 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830365 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-home" (OuterVolumeSpecName: "home") pod "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" (UID: "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:52:50.830453 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830388 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-model-cache" (OuterVolumeSpecName: "model-cache") pod "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" (UID: "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:52:50.830562 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830512 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" (UID: "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:52:50.830562 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830531 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-home\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:52:50.830562 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.830550 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-model-cache\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:52:50.832291 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.832260 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" (UID: "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:52:50.832405 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.832318 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-dshm" (OuterVolumeSpecName: "dshm") pod "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" (UID: "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:52:50.832405 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.832349 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-kube-api-access-6vwjp" (OuterVolumeSpecName: "kube-api-access-6vwjp") pod "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" (UID: "817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76"). InnerVolumeSpecName "kube-api-access-6vwjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:52:50.931308 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.931271 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-dshm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:52:50.931308 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.931301 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwjp\" (UniqueName: \"kubernetes.io/projected/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-kube-api-access-6vwjp\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:52:50.931308 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.931313 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tmp-dir\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:52:50.931540 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.931322 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76-tls-certs\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:52:50.939821 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.939789 2572 generic.go:356] "Generic (PLEG): container finished" podID="817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" containerID="9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c" exitCode=0 Jul 03 15:52:50.939951 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.939831 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" Jul 03 15:52:50.939951 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.939879 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" event={"ID":"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76","Type":"ContainerDied","Data":"9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c"} Jul 03 15:52:50.939951 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.939922 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z" event={"ID":"817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76","Type":"ContainerDied","Data":"006ebcc0baefb7fd601459479545b61b31dd93670da3c360b8e21a8bb690de06"} Jul 03 15:52:50.939951 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.939941 2572 scope.go:117] "RemoveContainer" containerID="9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c" Jul 03 15:52:50.948982 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.948963 2572 scope.go:117] "RemoveContainer" containerID="9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c" Jul 03 15:52:50.949268 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:52:50.949249 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c\": container with ID starting with 9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c not found: ID does not exist" containerID="9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c" Jul 03 15:52:50.949328 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.949278 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c"} err="failed to get container status \"9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c\": rpc error: code = NotFound desc = could not find container \"9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c\": container with ID starting with 9f193aee769029491468936b6727632b0c91782bf9c9f0f3bf15179f623a1e3c not found: ID does not exist" Jul 03 15:52:50.960079 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.960051 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z"] Jul 03 15:52:50.963530 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:50.963510 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/sub-test-model-31658c-kserve-5f6f9f8cf-tgc4z"] Jul 03 15:52:52.231029 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:52.230976 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" path="/var/lib/kubelet/pods/817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76/volumes" Jul 03 15:52:55.819592 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:55.819549 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:55.820011 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:55.819615 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:55.831922 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:55.831900 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:52:55.977631 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:52:55.977600 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:53:17.581524 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:17.581499 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:53:17.585820 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:17.585803 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bzpv8_b762f10c-f840-4d77-9597-e77cc779b7f1/ovn-acl-logging/0.log" Jul 03 15:53:41.298535 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.298460 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs"] Jul 03 15:53:41.298885 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.298743 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" containerName="main" Jul 03 15:53:41.298885 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.298753 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" containerName="main" Jul 03 15:53:41.298885 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.298805 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="817ea2f2-c7e8-45ba-ae7f-117ac4f9bd76" containerName="main" Jul 03 15:53:41.302331 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.302302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.307186 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.307159 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"openshift-service-ca.crt\"" Jul 03 15:53:41.307315 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.307199 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"default-dockercfg-bz8cf\"" Jul 03 15:53:41.307315 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.307244 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"rate-test-model-31658c-kserve-self-signed-certs\"" Jul 03 15:53:41.307315 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.307204 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-31658c\"/\"kube-root-ca.crt\"" Jul 03 15:53:41.312286 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.312262 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs"] Jul 03 15:53:41.364716 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.364681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tmp-dir\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.364716 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.364718 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-dshm\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.364964 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.364760 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jbr\" (UniqueName: \"kubernetes.io/projected/68b1b6a2-6189-4fa0-ae84-6cbc63556891-kube-api-access-j7jbr\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.364964 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.364780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-home\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.364964 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.364796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-model-cache\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.364964 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.364818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tls-certs\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466362 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466329 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tls-certs\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466579 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466391 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tmp-dir\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466579 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-dshm\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466579 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jbr\" (UniqueName: \"kubernetes.io/projected/68b1b6a2-6189-4fa0-ae84-6cbc63556891-kube-api-access-j7jbr\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466579 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466478 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-home\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466579 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-model-cache\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466860 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466834 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tmp-dir\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466916 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466878 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-model-cache\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.466916 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.466907 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-home\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.468773 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.468753 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-dshm\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.469092 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.469071 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tls-certs\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.474763 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.474735 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jbr\" (UniqueName: \"kubernetes.io/projected/68b1b6a2-6189-4fa0-ae84-6cbc63556891-kube-api-access-j7jbr\") pod \"rate-test-model-31658c-kserve-69cbb44c5-mqqgs\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.613178 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.613148 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:41.748652 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:41.748627 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs"] Jul 03 15:53:41.750643 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:53:41.750614 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68b1b6a2_6189_4fa0_ae84_6cbc63556891.slice/crio-12b7c60a9c076b46fdd4ddd1c820925eefd78fd71038ee14b78849e3b3092db8 WatchSource:0}: Error finding container 12b7c60a9c076b46fdd4ddd1c820925eefd78fd71038ee14b78849e3b3092db8: Status 404 returned error can't find the container with id 12b7c60a9c076b46fdd4ddd1c820925eefd78fd71038ee14b78849e3b3092db8 Jul 03 15:53:42.180238 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:42.180194 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" event={"ID":"68b1b6a2-6189-4fa0-ae84-6cbc63556891","Type":"ContainerStarted","Data":"aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac"} Jul 03 15:53:42.180238 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:42.180241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" event={"ID":"68b1b6a2-6189-4fa0-ae84-6cbc63556891","Type":"ContainerStarted","Data":"12b7c60a9c076b46fdd4ddd1c820925eefd78fd71038ee14b78849e3b3092db8"} Jul 03 15:53:42.197774 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:42.197713 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" podStartSLOduration=1.197696631 podStartE2EDuration="1.197696631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:53:42.195273566 +0000 UTC m=+2426.427706455" watchObservedRunningTime="2026-07-03 15:53:42.197696631 +0000 UTC m=+2426.430129520" Jul 03 15:53:51.614042 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:51.613969 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:51.614042 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:51.614060 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:51.626901 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:51.626875 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:53:52.245757 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:53:52.245728 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:54:20.216795 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:20.216757 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz"] Jul 03 15:54:21.073364 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:21.073329 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs"] Jul 03 15:54:35.236697 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.236649 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" podUID="64ca9f41-9940-42e0-a284-8d4aad3784b5" containerName="main" containerID="cri-o://cb0b9dc42311dfdd7c7101707947064af1dfbb96953c998cfc3774407fa3fd0a" gracePeriod=45 Jul 03 15:54:35.432161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.432118 2572 generic.go:356] "Generic (PLEG): container finished" podID="64ca9f41-9940-42e0-a284-8d4aad3784b5" containerID="cb0b9dc42311dfdd7c7101707947064af1dfbb96953c998cfc3774407fa3fd0a" exitCode=0 Jul 03 15:54:35.432314 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.432164 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" event={"ID":"64ca9f41-9940-42e0-a284-8d4aad3784b5","Type":"ContainerDied","Data":"cb0b9dc42311dfdd7c7101707947064af1dfbb96953c998cfc3774407fa3fd0a"} Jul 03 15:54:35.483625 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.483597 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:54:35.631731 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.631701 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7frrc\" (UniqueName: \"kubernetes.io/projected/64ca9f41-9940-42e0-a284-8d4aad3784b5-kube-api-access-7frrc\") pod \"64ca9f41-9940-42e0-a284-8d4aad3784b5\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " Jul 03 15:54:35.631731 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.631735 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-model-cache\") pod \"64ca9f41-9940-42e0-a284-8d4aad3784b5\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " Jul 03 15:54:35.631957 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.631760 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ca9f41-9940-42e0-a284-8d4aad3784b5-tls-certs\") pod \"64ca9f41-9940-42e0-a284-8d4aad3784b5\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " Jul 03 15:54:35.631957 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.631830 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-dshm\") pod \"64ca9f41-9940-42e0-a284-8d4aad3784b5\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " Jul 03 15:54:35.631957 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.631846 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-home\") pod \"64ca9f41-9940-42e0-a284-8d4aad3784b5\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " Jul 03 15:54:35.631957 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.631890 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-tmp-dir\") pod \"64ca9f41-9940-42e0-a284-8d4aad3784b5\" (UID: \"64ca9f41-9940-42e0-a284-8d4aad3784b5\") " Jul 03 15:54:35.632200 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.632040 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "64ca9f41-9940-42e0-a284-8d4aad3784b5" (UID: "64ca9f41-9940-42e0-a284-8d4aad3784b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:35.632200 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.632175 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-model-cache\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:35.632285 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.632193 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-home" (OuterVolumeSpecName: "home") pod "64ca9f41-9940-42e0-a284-8d4aad3784b5" (UID: "64ca9f41-9940-42e0-a284-8d4aad3784b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:35.632285 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.632244 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64ca9f41-9940-42e0-a284-8d4aad3784b5" (UID: "64ca9f41-9940-42e0-a284-8d4aad3784b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:35.633973 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.633947 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-dshm" (OuterVolumeSpecName: "dshm") pod "64ca9f41-9940-42e0-a284-8d4aad3784b5" (UID: "64ca9f41-9940-42e0-a284-8d4aad3784b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:35.634088 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.634038 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ca9f41-9940-42e0-a284-8d4aad3784b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64ca9f41-9940-42e0-a284-8d4aad3784b5" (UID: "64ca9f41-9940-42e0-a284-8d4aad3784b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:54:35.634088 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.634049 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ca9f41-9940-42e0-a284-8d4aad3784b5-kube-api-access-7frrc" (OuterVolumeSpecName: "kube-api-access-7frrc") pod "64ca9f41-9940-42e0-a284-8d4aad3784b5" (UID: "64ca9f41-9940-42e0-a284-8d4aad3784b5"). InnerVolumeSpecName "kube-api-access-7frrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:54:35.733538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.733485 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-dshm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:35.733538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.733528 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-home\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:35.733538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.733537 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ca9f41-9940-42e0-a284-8d4aad3784b5-tmp-dir\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:35.733538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.733548 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frrc\" (UniqueName: \"kubernetes.io/projected/64ca9f41-9940-42e0-a284-8d4aad3784b5-kube-api-access-7frrc\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:35.733538 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:35.733557 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ca9f41-9940-42e0-a284-8d4aad3784b5-tls-certs\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:36.092937 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.092900 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" podUID="68b1b6a2-6189-4fa0-ae84-6cbc63556891" containerName="main" containerID="cri-o://aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac" gracePeriod=45 Jul 03 15:54:36.343014 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.342936 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:54:36.437917 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.437885 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" Jul 03 15:54:36.438138 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.437926 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz" event={"ID":"64ca9f41-9940-42e0-a284-8d4aad3784b5","Type":"ContainerDied","Data":"68c22c226efec0bb5f0f039469b4fdb4d5c8fefdf54a84c90b57ca5450573931"} Jul 03 15:54:36.438138 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.437977 2572 scope.go:117] "RemoveContainer" containerID="cb0b9dc42311dfdd7c7101707947064af1dfbb96953c998cfc3774407fa3fd0a" Jul 03 15:54:36.438261 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438185 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tmp-dir\") pod \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " Jul 03 15:54:36.438261 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438215 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-dshm\") pod \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " Jul 03 15:54:36.438261 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438246 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tls-certs\") pod \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " Jul 03 15:54:36.438413 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438278 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-model-cache\") pod \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " Jul 03 15:54:36.438413 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438393 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-home\") pod \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " Jul 03 15:54:36.438920 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438428 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7jbr\" (UniqueName: \"kubernetes.io/projected/68b1b6a2-6189-4fa0-ae84-6cbc63556891-kube-api-access-j7jbr\") pod \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\" (UID: \"68b1b6a2-6189-4fa0-ae84-6cbc63556891\") " Jul 03 15:54:36.438920 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438483 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68b1b6a2-6189-4fa0-ae84-6cbc63556891" (UID: "68b1b6a2-6189-4fa0-ae84-6cbc63556891"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:36.438920 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438623 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tmp-dir\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:36.438920 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438699 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-model-cache" (OuterVolumeSpecName: "model-cache") pod "68b1b6a2-6189-4fa0-ae84-6cbc63556891" (UID: "68b1b6a2-6189-4fa0-ae84-6cbc63556891"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:36.438920 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.438712 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-home" (OuterVolumeSpecName: "home") pod "68b1b6a2-6189-4fa0-ae84-6cbc63556891" (UID: "68b1b6a2-6189-4fa0-ae84-6cbc63556891"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:36.439829 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.439802 2572 generic.go:356] "Generic (PLEG): container finished" podID="68b1b6a2-6189-4fa0-ae84-6cbc63556891" containerID="aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac" exitCode=0 Jul 03 15:54:36.439949 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.439864 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" Jul 03 15:54:36.440050 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.439936 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" event={"ID":"68b1b6a2-6189-4fa0-ae84-6cbc63556891","Type":"ContainerDied","Data":"aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac"} Jul 03 15:54:36.440050 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.439978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs" event={"ID":"68b1b6a2-6189-4fa0-ae84-6cbc63556891","Type":"ContainerDied","Data":"12b7c60a9c076b46fdd4ddd1c820925eefd78fd71038ee14b78849e3b3092db8"} Jul 03 15:54:36.440973 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.440846 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-dshm" (OuterVolumeSpecName: "dshm") pod "68b1b6a2-6189-4fa0-ae84-6cbc63556891" (UID: "68b1b6a2-6189-4fa0-ae84-6cbc63556891"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 15:54:36.441124 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.441105 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68b1b6a2-6189-4fa0-ae84-6cbc63556891" (UID: "68b1b6a2-6189-4fa0-ae84-6cbc63556891"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 15:54:36.441186 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.441120 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b1b6a2-6189-4fa0-ae84-6cbc63556891-kube-api-access-j7jbr" (OuterVolumeSpecName: "kube-api-access-j7jbr") pod "68b1b6a2-6189-4fa0-ae84-6cbc63556891" (UID: "68b1b6a2-6189-4fa0-ae84-6cbc63556891"). InnerVolumeSpecName "kube-api-access-j7jbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 15:54:36.456319 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.456301 2572 scope.go:117] "RemoveContainer" containerID="aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac" Jul 03 15:54:36.464622 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.464602 2572 scope.go:117] "RemoveContainer" containerID="aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac" Jul 03 15:54:36.464915 ip-10-0-143-112 kubenswrapper[2572]: E0703 15:54:36.464893 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac\": container with ID starting with aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac not found: ID does not exist" containerID="aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac" Jul 03 15:54:36.464981 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.464923 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac"} err="failed to get container status \"aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac\": rpc error: code = NotFound desc = could not find container \"aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac\": container with ID starting with aa94429f89a2915c379e3e63cfd59d2d40792dda2731b4464ef5a1d9881c88ac not found: ID does not exist" Jul 03 15:54:36.465313 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.465294 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz"] Jul 03 15:54:36.468125 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.468102 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3703d4/rate-test-model-3703d4-kserve-845687f9ff-ghjsz"] Jul 03 15:54:36.539445 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.539405 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-home\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:36.539445 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.539438 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7jbr\" (UniqueName: \"kubernetes.io/projected/68b1b6a2-6189-4fa0-ae84-6cbc63556891-kube-api-access-j7jbr\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:36.539445 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.539450 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-dshm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:36.539667 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.539461 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b1b6a2-6189-4fa0-ae84-6cbc63556891-tls-certs\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:36.539667 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.539471 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b1b6a2-6189-4fa0-ae84-6cbc63556891-model-cache\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\"" Jul 03 15:54:36.761652 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.761623 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs"] Jul 03 15:54:36.763328 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:36.763309 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-31658c/rate-test-model-31658c-kserve-69cbb44c5-mqqgs"] Jul 03 15:54:38.230408 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:38.230369 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ca9f41-9940-42e0-a284-8d4aad3784b5" path="/var/lib/kubelet/pods/64ca9f41-9940-42e0-a284-8d4aad3784b5/volumes" Jul 03 15:54:38.230784 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:54:38.230765 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b1b6a2-6189-4fa0-ae84-6cbc63556891" path="/var/lib/kubelet/pods/68b1b6a2-6189-4fa0-ae84-6cbc63556891/volumes" Jul 03 15:55:12.770610 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:12.770570 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54b68d65c6-dh6nt_938e32af-521a-404e-aac5-0b27a829117e/authorino/0.log" Jul 03 15:55:18.336143 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:18.336111 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hvhs7_9ef602d3-50e5-4bc2-a02e-488c67bf14cf/manager/0.log" Jul 03 15:55:18.458511 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:18.458480 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86b59fc499-f5wg5_36340a31-6ebd-40b8-84f3-430c400a0091/manager/0.log" Jul 03 15:55:18.465437 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:18.465415 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86b59fc499-f5wg5_36340a31-6ebd-40b8-84f3-430c400a0091/copy-manifests/0.log" Jul 03 15:55:19.602289 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.602258 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs_1978064d-027f-4af4-947a-75c8e46f7e98/util/0.log" Jul 03 15:55:19.609567 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.609538 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs_1978064d-027f-4af4-947a-75c8e46f7e98/pull/0.log" Jul 03 15:55:19.615103 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.615078 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs_1978064d-027f-4af4-947a-75c8e46f7e98/extract/0.log" Jul 03 15:55:19.732192 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.732161 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf_bb354ebe-e866-470f-b7d7-aed008e1cee8/extract/0.log" Jul 03 15:55:19.738851 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.738824 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf_bb354ebe-e866-470f-b7d7-aed008e1cee8/util/0.log" Jul 03 15:55:19.747618 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.747593 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf_bb354ebe-e866-470f-b7d7-aed008e1cee8/pull/0.log" Jul 03 15:55:19.857694 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.857611 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b_d383f0b3-a2ea-4807-ad91-22139105766d/util/0.log" Jul 03 15:55:19.863695 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.863670 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b_d383f0b3-a2ea-4807-ad91-22139105766d/pull/0.log" Jul 03 15:55:19.869254 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.869229 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b_d383f0b3-a2ea-4807-ad91-22139105766d/extract/0.log" Jul 03 15:55:19.983933 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.983905 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj_e97f5a4b-c499-46e7-a37e-76683bf3aa6a/util/0.log" Jul 03 15:55:19.994233 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:19.994204 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj_e97f5a4b-c499-46e7-a37e-76683bf3aa6a/pull/0.log" Jul 03 15:55:20.002178 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:20.002154 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj_e97f5a4b-c499-46e7-a37e-76683bf3aa6a/extract/0.log" Jul 03 15:55:20.127289 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:20.127207 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54b68d65c6-dh6nt_938e32af-521a-404e-aac5-0b27a829117e/authorino/0.log" Jul 03 15:55:21.305362 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:21.305330 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678gfsbj_cfbdb13f-19b4-4460-8655-616ce959226a/istio-proxy/0.log" Jul 03 15:55:21.547972 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:21.547939 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d497994-8fhn2_0b42884e-e056-4bc3-85f7-d2879960b3fb/kube-auth-proxy/0.log" Jul 03 15:55:22.639964 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:22.639920 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-cctf8_1094313a-05ef-4888-bdeb-c9a45a66fabb/main/0.log" Jul 03 15:55:22.887833 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:22.887790 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb26cq_0ee657f9-a711-4168-a7c1-f66e8522a306/main/0.log" Jul 03 15:55:27.193435 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.193402 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2795n/must-gather-2jmld"] Jul 03 15:55:27.193829 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.193685 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b1b6a2-6189-4fa0-ae84-6cbc63556891" containerName="main" Jul 03 15:55:27.193829 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.193697 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b1b6a2-6189-4fa0-ae84-6cbc63556891" containerName="main" Jul 03 15:55:27.193829 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.193709 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ca9f41-9940-42e0-a284-8d4aad3784b5" containerName="main" Jul 03 15:55:27.193829 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.193715 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ca9f41-9940-42e0-a284-8d4aad3784b5" containerName="main" Jul 03 15:55:27.193829 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.193764 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b1b6a2-6189-4fa0-ae84-6cbc63556891" containerName="main" Jul 03 15:55:27.193829 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.193773 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ca9f41-9940-42e0-a284-8d4aad3784b5" containerName="main" Jul 03 15:55:27.199067 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.197429 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.200776 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.200745 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2795n\"/\"default-dockercfg-t2c96\"" Jul 03 15:55:27.200917 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.200752 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2795n\"/\"openshift-service-ca.crt\"" Jul 03 15:55:27.200917 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.200807 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2795n\"/\"kube-root-ca.crt\"" Jul 03 15:55:27.209598 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.209576 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2795n/must-gather-2jmld"] Jul 03 15:55:27.251471 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.251433 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754abce5-2261-4a27-a210-86c16df6f2aa-must-gather-output\") pod \"must-gather-2jmld\" (UID: \"754abce5-2261-4a27-a210-86c16df6f2aa\") " pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.251658 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.251489 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52n8\" (UniqueName: \"kubernetes.io/projected/754abce5-2261-4a27-a210-86c16df6f2aa-kube-api-access-p52n8\") pod \"must-gather-2jmld\" (UID: \"754abce5-2261-4a27-a210-86c16df6f2aa\") " pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.352559 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.352462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52n8\" (UniqueName: \"kubernetes.io/projected/754abce5-2261-4a27-a210-86c16df6f2aa-kube-api-access-p52n8\") pod \"must-gather-2jmld\" (UID: \"754abce5-2261-4a27-a210-86c16df6f2aa\") " pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.352559 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.352535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754abce5-2261-4a27-a210-86c16df6f2aa-must-gather-output\") pod \"must-gather-2jmld\" (UID: \"754abce5-2261-4a27-a210-86c16df6f2aa\") " pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.352831 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.352814 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754abce5-2261-4a27-a210-86c16df6f2aa-must-gather-output\") pod \"must-gather-2jmld\" (UID: \"754abce5-2261-4a27-a210-86c16df6f2aa\") " pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.367397 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.367366 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52n8\" (UniqueName: \"kubernetes.io/projected/754abce5-2261-4a27-a210-86c16df6f2aa-kube-api-access-p52n8\") pod \"must-gather-2jmld\" (UID: \"754abce5-2261-4a27-a210-86c16df6f2aa\") " pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.508766 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.508735 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2795n/must-gather-2jmld" Jul 03 15:55:27.640860 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.640832 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2795n/must-gather-2jmld"] Jul 03 15:55:27.642864 ip-10-0-143-112 kubenswrapper[2572]: W0703 15:55:27.642826 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod754abce5_2261_4a27_a210_86c16df6f2aa.slice/crio-09cbbbdce444ca4ad3c04b0410aad0fbcd3d96bb3d83612b5d3c9a315bceb6d0 WatchSource:0}: Error finding container 09cbbbdce444ca4ad3c04b0410aad0fbcd3d96bb3d83612b5d3c9a315bceb6d0: Status 404 returned error can't find the container with id 09cbbbdce444ca4ad3c04b0410aad0fbcd3d96bb3d83612b5d3c9a315bceb6d0 Jul 03 15:55:27.644698 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.644682 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 15:55:27.688045 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:27.688007 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2795n/must-gather-2jmld" event={"ID":"754abce5-2261-4a27-a210-86c16df6f2aa","Type":"ContainerStarted","Data":"09cbbbdce444ca4ad3c04b0410aad0fbcd3d96bb3d83612b5d3c9a315bceb6d0"} Jul 03 15:55:28.695369 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:28.695333 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2795n/must-gather-2jmld" event={"ID":"754abce5-2261-4a27-a210-86c16df6f2aa","Type":"ContainerStarted","Data":"183170c407d578f105b5ff32159871a1ec445167787679e85579397074fe361b"} Jul 03 15:55:28.695369 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:28.695374 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2795n/must-gather-2jmld" event={"ID":"754abce5-2261-4a27-a210-86c16df6f2aa","Type":"ContainerStarted","Data":"93fb3b8cafec0b73cde131ca262dcd4c5bb1f0a39d94e2054be21a5bd4fac1d3"} Jul 03 15:55:29.783793 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:29.783759 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c29qj_8bb9d8dd-f7d0-4f5a-be41-ce3a3ab13bec/global-pull-secret-syncer/0.log" Jul 03 15:55:29.960439 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:29.960405 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v62c6_03c15373-89c7-4c47-8c01-0c6498dedce5/konnectivity-agent/0.log" Jul 03 15:55:30.048940 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:30.048858 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-112.ec2.internal_79a3a725957a215de175bf549eccdfcf/haproxy/0.log" Jul 03 15:55:34.802439 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.802392 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs_1978064d-027f-4af4-947a-75c8e46f7e98/extract/0.log" Jul 03 15:55:34.824392 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.824366 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs_1978064d-027f-4af4-947a-75c8e46f7e98/util/0.log" Jul 03 15:55:34.841455 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.841425 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598kbqs_1978064d-027f-4af4-947a-75c8e46f7e98/pull/0.log" Jul 03 15:55:34.869490 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.869457 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf_bb354ebe-e866-470f-b7d7-aed008e1cee8/extract/0.log" Jul 03 15:55:34.886138 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.886107 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf_bb354ebe-e866-470f-b7d7-aed008e1cee8/util/0.log" Jul 03 15:55:34.904088 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.904055 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0827hf_bb354ebe-e866-470f-b7d7-aed008e1cee8/pull/0.log" Jul 03 15:55:34.931304 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.931272 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b_d383f0b3-a2ea-4807-ad91-22139105766d/extract/0.log" Jul 03 15:55:34.945934 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.945905 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b_d383f0b3-a2ea-4807-ad91-22139105766d/util/0.log" Jul 03 15:55:34.968021 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.967974 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hvq8b_d383f0b3-a2ea-4807-ad91-22139105766d/pull/0.log" Jul 03 15:55:34.992674 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:34.992648 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj_e97f5a4b-c499-46e7-a37e-76683bf3aa6a/extract/0.log" Jul 03 15:55:35.013531 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:35.013507 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj_e97f5a4b-c499-46e7-a37e-76683bf3aa6a/util/0.log" Jul 03 15:55:35.034583 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:35.034546 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14p4jj_e97f5a4b-c499-46e7-a37e-76683bf3aa6a/pull/0.log" Jul 03 15:55:35.468849 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:35.468815 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54b68d65c6-dh6nt_938e32af-521a-404e-aac5-0b27a829117e/authorino/0.log" Jul 03 15:55:37.815144 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:37.815115 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t96ln_702b7539-b6d5-405a-8142-93b83e360345/node-exporter/0.log" Jul 03 15:55:37.833279 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:37.833246 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t96ln_702b7539-b6d5-405a-8142-93b83e360345/kube-rbac-proxy/0.log" Jul 03 15:55:37.849889 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:37.849861 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t96ln_702b7539-b6d5-405a-8142-93b83e360345/init-textfile/0.log" Jul 03 15:55:40.144029 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.143939 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2795n/must-gather-2jmld" podStartSLOduration=12.550844751 podStartE2EDuration="13.143914741s" totalImagesPullingTime="593.06999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 15:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:55:28.714050258 +0000 UTC m=+2532.946483148" watchObservedRunningTime="2026-07-03 15:55:40.143914741 +0000 UTC m=+2544.376347632" Jul 03 15:55:40.146455 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.146423 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg"] Jul 03 15:55:40.156535 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.156509 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.160494 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.160463 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg"] Jul 03 15:55:40.296062 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.296021 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-sys\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.296263 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.296095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-lib-modules\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.296263 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.296202 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-podres\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.296390 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.296315 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-proc\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.296390 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.296339 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hznj\" (UniqueName: \"kubernetes.io/projected/823262c3-38a1-4299-8125-d3f00cc1b971-kube-api-access-2hznj\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397142 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-proc\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397142 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397095 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hznj\" (UniqueName: \"kubernetes.io/projected/823262c3-38a1-4299-8125-d3f00cc1b971-kube-api-access-2hznj\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-sys\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397164 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-proc\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-lib-modules\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397227 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-podres\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397231 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-sys\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397391 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397376 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-lib-modules\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.397705 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.397391 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/823262c3-38a1-4299-8125-d3f00cc1b971-podres\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.406453 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.406422 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hznj\" (UniqueName: \"kubernetes.io/projected/823262c3-38a1-4299-8125-d3f00cc1b971-kube-api-access-2hznj\") pod \"perf-node-gather-daemonset-qksvg\" (UID: \"823262c3-38a1-4299-8125-d3f00cc1b971\") " pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.474490 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.474455 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:40.671061 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.670104 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg"] Jul 03 15:55:40.778746 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:40.777339 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" event={"ID":"823262c3-38a1-4299-8125-d3f00cc1b971","Type":"ContainerStarted","Data":"46e77b4b2256d00d8e2a1ce3ea52a13ca70c4920aa6c8f232b3747f7e55eb6b8"} Jul 03 15:55:41.785060 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:41.785021 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" event={"ID":"823262c3-38a1-4299-8125-d3f00cc1b971","Type":"ContainerStarted","Data":"58f77f9af5b87931320d5ac5d5b52976aad912fd83da53d43164db3a091df656"} Jul 03 15:55:41.785458 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:41.785150 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:41.800026 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:41.799947 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" podStartSLOduration=1.799927218 podStartE2EDuration="1.799927218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 15:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 15:55:41.799585774 +0000 UTC m=+2546.032018663" watchObservedRunningTime="2026-07-03 15:55:41.799927218 +0000 UTC m=+2546.032360111" Jul 03 15:55:42.350234 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:42.350200 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2wrgx_0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83/dns/0.log" Jul 03 15:55:42.370129 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:42.370096 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2wrgx_0d91cdbf-9e99-4ebe-adcd-ea3d7adf1b83/kube-rbac-proxy/0.log" Jul 03 15:55:42.468505 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:42.468469 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq5dz_82d6aa1f-2b29-4328-83e5-2e961402a35f/dns-node-resolver/0.log" Jul 03 15:55:43.017301 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:43.017266 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-646bbcd79b-9792s_9413fe8e-3dfb-4502-be04-d26b3cae42ba/registry/0.log" Jul 03 15:55:43.074426 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:43.074393 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lct2j_b2dc4ba3-f734-4053-acee-fed33f36b791/node-ca/0.log" Jul 03 15:55:43.961161 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:43.961132 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678gfsbj_cfbdb13f-19b4-4460-8655-616ce959226a/istio-proxy/0.log" Jul 03 15:55:44.176068 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:44.176036 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d497994-8fhn2_0b42884e-e056-4bc3-85f7-d2879960b3fb/kube-auth-proxy/0.log" Jul 03 15:55:45.039207 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:45.039171 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmjkm_a97e8698-dc26-4b82-9bfb-60c7b176c487/serve-healthcheck-canary/0.log" Jul 03 15:55:45.571158 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:45.571123 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpqg8_8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8/kube-rbac-proxy/0.log" Jul 03 15:55:45.587796 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:45.587665 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpqg8_8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8/exporter/0.log" Jul 03 15:55:45.604882 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:45.604847 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kpqg8_8f5f109c-b88f-4b1c-aa9f-4d8615abd2c8/extractor/0.log" Jul 03 15:55:47.809006 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:47.808936 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2795n/perf-node-gather-daemonset-qksvg" Jul 03 15:55:48.253076 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:48.253019 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hvhs7_9ef602d3-50e5-4bc2-a02e-488c67bf14cf/manager/0.log" Jul 03 15:55:48.277389 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:48.277352 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86b59fc499-f5wg5_36340a31-6ebd-40b8-84f3-430c400a0091/manager/0.log" Jul 03 15:55:48.294573 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:48.294538 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86b59fc499-f5wg5_36340a31-6ebd-40b8-84f3-430c400a0091/copy-manifests/0.log" Jul 03 15:55:50.029724 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:50.029692 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c796b4fcc-d2pfs_938eb9a5-dd24-4749-8ecd-8ced9aec7901/manager/0.log" Jul 03 15:55:50.075221 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:50.075173 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-z7m47_e3d54199-d2d9-4640-aa3b-8a7d24c70c9c/openshift-lws-operator/0.log" Jul 03 15:55:50.491398 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:50.491362 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678gfsbj_cfbdb13f-19b4-4460-8655-616ce959226a/istio-proxy/0.log" Jul 03 15:55:50.694877 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:50.694844 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d497994-8fhn2_0b42884e-e056-4bc3-85f7-d2879960b3fb/kube-auth-proxy/0.log" Jul 03 15:55:52.109063 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:52.108946 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678gfsbj_cfbdb13f-19b4-4460-8655-616ce959226a/istio-proxy/0.log" Jul 03 15:55:52.325371 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:52.325289 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d497994-8fhn2_0b42884e-e056-4bc3-85f7-d2879960b3fb/kube-auth-proxy/0.log" Jul 03 15:55:56.327778 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.327750 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk58z_5e33c5cd-6688-4a91-9744-a36058fdb15f/kube-multus-additional-cni-plugins/0.log" Jul 03 15:55:56.343236 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.343211 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk58z_5e33c5cd-6688-4a91-9744-a36058fdb15f/egress-router-binary-copy/0.log" Jul 03 15:55:56.359029 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.359001 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk58z_5e33c5cd-6688-4a91-9744-a36058fdb15f/cni-plugins/0.log" Jul 03 15:55:56.376566 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.376541 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk58z_5e33c5cd-6688-4a91-9744-a36058fdb15f/bond-cni-plugin/0.log" Jul 03 15:55:56.395046 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.395018 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk58z_5e33c5cd-6688-4a91-9744-a36058fdb15f/routeoverride-cni/0.log" Jul 03 15:55:56.412577 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.412542 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk58z_5e33c5cd-6688-4a91-9744-a36058fdb15f/whereabouts-cni-bincopy/0.log" Jul 03 15:55:56.428613 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.428589 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jk58z_5e33c5cd-6688-4a91-9744-a36058fdb15f/whereabouts-cni/0.log" Jul 03 15:55:56.587514 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.587485 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gxxq5_220c038d-e52f-47b6-ab0a-3782911b6f83/kube-multus/0.log" Jul 03 15:55:56.736452 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.736422 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rjszq_6eb9e801-5990-47ae-964f-6f8e509cffbf/network-metrics-daemon/0.log" Jul 03 15:55:56.749653 ip-10-0-143-112 kubenswrapper[2572]: I0703 15:55:56.749619 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rjszq_6eb9e801-5990-47ae-964f-6f8e509cffbf/kube-rbac-proxy/0.log"