Jul 01 19:22:12.287284 ip-10-0-143-50 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 19:22:12.640036 ip-10-0-143-50 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:22:12.640036 ip-10-0-143-50 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 19:22:12.640036 ip-10-0-143-50 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:22:12.640036 ip-10-0-143-50 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 19:22:12.640036 ip-10-0-143-50 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:22:12.640926 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.640807    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 19:22:12.644827 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644809    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:12.644827 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644827    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644832    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644835    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644838    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644841    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644844    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644848    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644851    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644858    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644863    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644867    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644871    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644875    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644878    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644881    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644884    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644887    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644891    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644893    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644896    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:12.644897 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644899    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644903    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644906    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644909    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644912    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644915    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644918    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644920    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644925    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644929    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644932    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644935    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644937    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644941    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644944    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644946    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644949    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644952    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644954    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644957    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:12.645490 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644960    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644963    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644966    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644969    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644971    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644974    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644976    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644979    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644981    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644984    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.644987    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645002    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645005    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645008    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645012    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645015    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645017    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645020    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645023    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645026    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:12.646011 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645029    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645031    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645034    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645037    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645040    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645043    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645046    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645049    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645051    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645056    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645058    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645062    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645064    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645067    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645070    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645073    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645076    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645078    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645081    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645084    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:12.646520 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645086    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645089    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645091    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645094    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645097    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645099    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645104    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645107    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645110    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645112    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645115    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645118    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645121    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645123    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645128    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645131    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645134    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.645137    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646644    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:12.647058 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646654    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646657    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646661    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646664    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646667    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646672    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646676    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646679    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646683    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646686    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646689    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646692    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646695    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646698    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646701    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646704    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646706    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646709    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646712    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646717    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:12.647830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646720    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646723    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646725    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646728    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646731    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646734    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646737    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646739    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646742    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646747    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646750    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646752    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646755    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646758    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646761    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646763    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646766    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646769    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646771    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646774    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:12.648616 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646777    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646781    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646785    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646789    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646794    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646797    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646801    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646804    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646807    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646810    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646813    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646816    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646819    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646821    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646824    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646827    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646830    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646832    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646835    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:12.649156 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646838    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646841    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646844    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646846    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646849    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646853    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646856    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646859    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646861    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646864    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646867    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646869    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646872    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646875    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646877    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646881    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646884    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646887    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646890    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646892    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:12.649672 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646895    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646897    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646900    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646903    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646906    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646909    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646911    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646914    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646916    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646919    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646921    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646924    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646927    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646929    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646932    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646934    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646937    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646940    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.646942    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:12.650197 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649180    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649191    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649195    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649199    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649202    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649206    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649210    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649213    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649216    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649219    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649222    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649226    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649229    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649232    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649234    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649238    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649240    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649243    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649246    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649249    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:12.650697 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649251    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649254    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649257    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649260    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649263    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649269    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649272    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649275    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649278    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649281    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649283    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649286    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649289    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649294    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649296    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649299    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649302    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649305    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649308    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:12.651261 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649311    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649314    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649316    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649319    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649322    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649325    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649327    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649330    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649334    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649338    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649341    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649344    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649347    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649350    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649353    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649357    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649361    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649364    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649367    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:12.651780 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649370    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649373    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649376    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649378    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649381    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649384    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649387    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649390    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649393    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649396    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649399    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649401    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649404    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649408    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649413    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649416    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649419    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649422    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649425    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649427    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:12.652283 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649430    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649433    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649435    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649438    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649442    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649445    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649447    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649450    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649453    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649456    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649459    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649461    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649464    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649467    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649469    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649474    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649478    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649481    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649484    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:12.652830 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649487    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.649490    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649565    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649574    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649581    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649585    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649590    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649594    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649598    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649606    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649610    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649614    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649618    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649622    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649626    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649629    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649632    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649636    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649639    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649643    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649649    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649652    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649655    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649658    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 19:22:12.653308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649662    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649670    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649674    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649677    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649681    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649685    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649689    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649692    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649696    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649699    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649703    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649706    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649709    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649712    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649716    2576 flags.go:64] FLAG: --enable-server="true"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649719    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649723    2576 flags.go:64] FLAG: --event-burst="100"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649726    2576 flags.go:64] FLAG: --event-qps="50"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649729    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649733    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649736    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649741    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649744    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649747    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649750    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 01 19:22:12.653893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649753    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649756    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649760    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649764    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649767    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649770    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649773    2576 flags.go:64] FLAG: --feature-gates=""
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649777    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649780    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649783    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649786    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649789    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649793    2576 flags.go:64] FLAG: --help="false"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649797    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649800    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649803    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649806    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649810    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649814    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649817    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649820    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649823    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649826    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 19:22:12.654550 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649829    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649832    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649835    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649838    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649841    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649844    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649847    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649850    2576 flags.go:64] FLAG: --lock-file=""
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649853    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649856    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649860    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649865    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649874    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649877    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649880    2576 flags.go:64] FLAG: --logging-format="text"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649883    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649887    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649890    2576 flags.go:64] FLAG: --manifest-url=""
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649893    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649898    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649901    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649906    2576 flags.go:64] FLAG: --max-pods="110"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649910    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649913    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649916    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 19:22:12.655192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649919    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649922    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649925    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649928    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649935    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649938    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649941    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649944    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649947    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649954    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649956    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649960    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649963    2576 flags.go:64] FLAG: --port="10250"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649966    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649970    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e2217a86d9796e06"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649973    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649976    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649979    2576 flags.go:64] FLAG: --register-node="true"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649983    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649986    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.649994    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650017    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650020    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650024    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650028    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 19:22:12.655863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650031    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650034    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650037    2576 flags.go:64] FLAG: --runonce="false"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650040    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650044    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650047    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650050    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650053    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650057    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650060    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650064    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650067    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650070    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650073    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650076    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650079    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650082    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650086    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650092    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650095    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650098    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650103    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650106    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650109    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650112    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 19:22:12.656585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650115    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650119    2576 flags.go:64] FLAG: --v="2"
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650124    2576 flags.go:64] FLAG: --version="false"
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650130    2576 flags.go:64] FLAG: --vmodule=""
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650134    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650138    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650272    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650278    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650282    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650285    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650288    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650292    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650295    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650298    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650302    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650305    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650308    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650311    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650313    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650316    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650319    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:12.657247 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650322    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650325    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650328    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650330    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650333    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650336    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650339    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650342    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650345    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650347    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650350    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650353    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650355    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650360    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650363    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650366    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650369    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650372    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650374    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:12.657783 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650377    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650380    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650383    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650386    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650388    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650391    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650394    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650397    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650400    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650402    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650405    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650408    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650410    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650413    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650416    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650419    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650421    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650424    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650427    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650429    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:12.658298 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650432    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650435    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650438    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650442    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650445    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650448    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650455    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650458    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650461    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650464    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650467    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650469    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650472    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650475    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650478    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650483    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650486    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650490    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650493    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:12.658860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650497    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650500    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650503    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650505    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650508    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650511    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650513    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650516    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650519    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650522    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650524    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650527    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650530    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650532    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650535    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650537    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650540    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650543    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650545    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:12.659399 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650550    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650553    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650555    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650558    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650562    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650565    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.650567    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.650573    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.657735    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.657750    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.658513    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.658521    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658584    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658590    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658594    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:12.659896 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658598    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658601    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658605    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658608    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658612    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658615    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658618    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658621    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658624    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658627    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658630    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658636    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658640    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658644    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658647    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658650    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658653    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658656    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658659    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658662    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:12.660309 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658664    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658668    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658671    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658674    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658677    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658680    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658684    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658687    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658690    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658693    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658696    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658699    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658702    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658705    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658708    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658711    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658714    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658717    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658719    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658722    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:12.660821 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658725    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658728    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658730    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658733    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658737    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658739    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658742    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658745    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658747    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658752    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658756    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658759    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658761    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658764    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658768    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658771    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658774    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658777    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658780    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:12.661368 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658784    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658786    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658789    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658792    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658795    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658798    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658801    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658804    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658807    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658810    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658814    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658817    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658821    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658825    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658828    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658831    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658833    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658836    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658839    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658841    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:12.661860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658844    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658847    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658849    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658852    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658855    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658858    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658861    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658864    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658869    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658872    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658875    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658878    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658880    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658883    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658886    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658889    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:12.662397 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.658891    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.658897    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659057    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659064    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659068    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659071    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659075    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659078    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659081    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659084    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659088    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659091    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659094    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659096    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659099    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659102    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:12.662845 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659104    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659107    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659111    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659116    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659119    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659122    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659125    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659127    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659130    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659135    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659138    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659141    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659143    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659146    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659149    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659151    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659154    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659157    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659160    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659163    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:12.663459 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659165    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659168    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659171    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659174    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659177    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659179    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659182    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659185    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659188    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659190    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659193    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659196    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659199    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659202    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659204    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659207    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659210    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659213    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659215    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659218    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:12.664107 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659221    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659224    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659228    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659231    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659233    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659238    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659241    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659244    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659248    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659250    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659253    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659256    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659258    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659261    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659264    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659267    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659269    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659272    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659275    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:12.664631 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659278    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659281    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659283    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659286    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659288    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659291    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659295    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659297    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659300    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659303    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659306    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659308    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659313    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659316    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659319    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659322    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659326    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659330    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659333    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:12.665417 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659335    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659339    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659341    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659344    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659347    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659350    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:12.659352    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.659357    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.659476    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.662395    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.663710    2576 server.go:1019] "Starting client certificate rotation"
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.663834    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.663887    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:22:12.666192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.666050    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:22:12.667335 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.667303    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:22:12.680164 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.680138    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 01 19:22:12.682646 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.682621    2576 log.go:25] "Validated CRI v1 image API"
Jul 01 19:22:12.684299 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.684281    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 19:22:12.689342 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.689323    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cb93c9f2-d75a-405c-8df4-ef9a96a0652c:/dev/nvme0n1p3 eb36adc5-3e08-4052-bb7d-9617db93a369:/dev/nvme0n1p4]
Jul 01 19:22:12.689418 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.689343    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 19:22:12.694677 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.694558    2576 manager.go:216] Machine: {Timestamp:2026-07-01 19:22:12.693312908 +0000 UTC m=+0.306356288 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091321 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d7eb52ab54715ee6d260f09e7dc35 SystemUUID:ec2d7eb5-2ab5-4715-ee6d-260f09e7dc35 BootID:991f6bb2-3dbc-4204-b188-ecb4da3b6bce Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8d:63:86:c7:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8d:63:86:c7:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:74:6f:3f:96:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 19:22:12.694677 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.694673    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 19:22:12.694806 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.694761    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 19:22:12.697705 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.697670    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 19:22:12.697860 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.697714    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 19:22:12.698592 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.698573    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 19:22:12.698629 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.698595    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 19:22:12.698629 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.698605    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:22:12.699182 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.699171    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:22:12.700110 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.700099    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 19:22:12.700986 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.700971    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:22:12.704448 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.704434    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:22:12.704576 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.704566    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 19:22:12.704609 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.704597    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 19:22:12.704638 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.704610    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 19:22:12.704638 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.704634    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 19:22:12.704692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.704648    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 01 19:22:12.704692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.704659    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 19:22:12.705074 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.705062    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:22:12.705135 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.705080    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:22:12.707163 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.707145    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 19:22:12.708622 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.708608    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 19:22:12.708682 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.708640    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 19:22:12.709461 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709449    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 19:22:12.709509 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709470    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 19:22:12.709509 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709479    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 19:22:12.709509 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709486    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 19:22:12.709509 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709496    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 19:22:12.709509 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709502    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 19:22:12.709509 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709511    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 19:22:12.709661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709517    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 19:22:12.709661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709527    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 19:22:12.709661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709536    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 19:22:12.709661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709560    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 19:22:12.709661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.709567    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 19:22:12.710218 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.710201    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 19:22:12.710218 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.710216    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 19:22:12.713696 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.713679    2576 server.go:1268] "Started kubelet"
Jul 01 19:22:12.713804 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.713761    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 19:22:12.713804 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.713780    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 19:22:12.713907 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.713867    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 19:22:12.714704 ip-10-0-143-50 systemd[1]: Started Kubernetes Kubelet.
Jul 01 19:22:12.718150 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.718123    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 19:22:12.718243 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.718193    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 19:22:12.718686 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.718663    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 19:22:12.719093 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.718959    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 19:22:12.719192 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.719090    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 19:22:12.719920 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.719888    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 19:22:12.721314 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.720485    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-50.ec2.internal.18be40c894755bff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-50.ec2.internal,UID:ip-10-0-143-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-50.ec2.internal,},FirstTimestamp:2026-07-01 19:22:12.713651199 +0000 UTC m=+0.326694581,LastTimestamp:2026-07-01 19:22:12.713651199 +0000 UTC m=+0.326694581,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-50.ec2.internal,}"
Jul 01 19:22:12.725399 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.725380    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 19:22:12.725485 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.725397    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 19:22:12.725549 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.725532    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:12.726103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726070    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 19:22:12.726206 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726176    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 19:22:12.726206 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726196    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 19:22:12.726359 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726345    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 19:22:12.726359 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726358    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 19:22:12.726745 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726724    2576 factory.go:153] Registering CRI-O factory
Jul 01 19:22:12.726745 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726748    2576 factory.go:223] Registration of the crio container factory successfully
Jul 01 19:22:12.726904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726807    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 19:22:12.726904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726818    2576 factory.go:55] Registering systemd factory
Jul 01 19:22:12.726904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726828    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 01 19:22:12.726904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726854    2576 factory.go:103] Registering Raw factory
Jul 01 19:22:12.726904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.726872    2576 manager.go:1196] Started watching for new ooms in manager
Jul 01 19:22:12.727607 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.727583    2576 manager.go:349] Starting recovery of all containers
Jul 01 19:22:12.727981 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.727955    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 19:22:12.728085 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.728065    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 19:22:12.738241 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.738087    2576 manager.go:354] Recovery completed
Jul 01 19:22:12.742699 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.742685    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:12.744664 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.744628    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:12.744736 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.744686    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:12.744736 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.744703    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:12.746442 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.746427    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:12.746532 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.746432    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 19:22:12.746532 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.746490    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 19:22:12.746532 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.746509    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:22:12.747225 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.747209    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:12.747294 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.747238    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:12.747294 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.747250    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:12.748897 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.748885    2576 policy_none.go:49] "None policy: Start"
Jul 01 19:22:12.748974 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.748902    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 19:22:12.748974 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.748913    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 19:22:12.751066 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.751054    2576 policy_none.go:47] "Start"
Jul 01 19:22:12.755675 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.755514    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-50.ec2.internal.18be40c8964e936b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-50.ec2.internal,UID:ip-10-0-143-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-50.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-50.ec2.internal,},FirstTimestamp:2026-07-01 19:22:12.744663915 +0000 UTC m=+0.357707298,LastTimestamp:2026-07-01 19:22:12.744663915 +0000 UTC m=+0.357707298,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-50.ec2.internal,}"
Jul 01 19:22:12.759647 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.759623    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hxcrt"
Jul 01 19:22:12.768524 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.768502    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hxcrt"
Jul 01 19:22:12.770191 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.770096    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-50.ec2.internal.18be40c8964f055b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-50.ec2.internal,UID:ip-10-0-143-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-50.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-50.ec2.internal,},FirstTimestamp:2026-07-01 19:22:12.744693083 +0000 UTC m=+0.357736465,LastTimestamp:2026-07-01 19:22:12.744693083 +0000 UTC m=+0.357736465,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-50.ec2.internal,}"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.787669    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.788874    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.788911    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.788933    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.788968    2576 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.789816    2576 manager.go:337] "Starting Device Plugin manager"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.789866    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.789881    2576 server.go:87] "Starting device plugin registration server"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.790190    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.790202    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.790339    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.790407    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.790414    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.791036    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.791093    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:12.803470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.791542    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:12.890440 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.890403    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:12.890440 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.890430    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal"]
Jul 01 19:22:12.890678 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.890529    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:12.891616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.891599    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:12.891712 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.891633    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:12.891712 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.891647    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:12.891712 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.891679    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.892218 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.892203    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:12.892284 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.892235    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:12.892284 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.892246    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:12.894956 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.894943    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:12.895109 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.895152 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895131    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:12.895763 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895737    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:12.895763 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895766    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:12.895906 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895777    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:12.895906 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895806    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:12.895906 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895830    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:12.895906 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.895841    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:12.898370 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.898354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.898439 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.898379    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:12.899204 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.899186    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:12.899204 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.899207    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:12.899337 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.899218    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:12.901881 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.901836    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.901881 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.901858    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-50.ec2.internal\": node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:12.924151 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.924122    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-50.ec2.internal\" not found" node="ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.926900 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.926879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e19a46725c74590d2419ed8e9a32c3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal\" (UID: \"7e19a46725c74590d2419ed8e9a32c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.927033 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.926915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e19a46725c74590d2419ed8e9a32c3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal\" (UID: \"7e19a46725c74590d2419ed8e9a32c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.927033 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:12.926957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef75bb6a2b1f0b788d8d4b4eec583ca4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-50.ec2.internal\" (UID: \"ef75bb6a2b1f0b788d8d4b4eec583ca4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.929046 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.929030    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-50.ec2.internal\" not found" node="ip-10-0-143-50.ec2.internal"
Jul 01 19:22:12.938063 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:12.938045    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:13.027246 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.027210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e19a46725c74590d2419ed8e9a32c3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal\" (UID: \"7e19a46725c74590d2419ed8e9a32c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.027246 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.027243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e19a46725c74590d2419ed8e9a32c3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal\" (UID: \"7e19a46725c74590d2419ed8e9a32c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.027476 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.027268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef75bb6a2b1f0b788d8d4b4eec583ca4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-50.ec2.internal\" (UID: \"ef75bb6a2b1f0b788d8d4b4eec583ca4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.027476 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.027315    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef75bb6a2b1f0b788d8d4b4eec583ca4-config\") pod \"kube-apiserver-proxy-ip-10-0-143-50.ec2.internal\" (UID: \"ef75bb6a2b1f0b788d8d4b4eec583ca4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.027476 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.027340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e19a46725c74590d2419ed8e9a32c3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal\" (UID: \"7e19a46725c74590d2419ed8e9a32c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.027476 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.027315    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e19a46725c74590d2419ed8e9a32c3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal\" (UID: \"7e19a46725c74590d2419ed8e9a32c3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.039248 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.039220    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:13.139650 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.139599    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:13.227871 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.227794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.233470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.233420    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.240134 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.240105    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:13.340467 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.340422    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:13.441541 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.441501    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:13.541810 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.541708    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-50.ec2.internal\" not found"
Jul 01 19:22:13.575164 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.575138    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:13.626286 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.626251    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.641447 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.641427    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:22:13.643026 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.643002    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal"
Jul 01 19:22:13.652998 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.652970    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:22:13.664368 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.664346    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 19:22:13.664498 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.664481    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 19:22:13.664545 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.664485    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 19:22:13.706073 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.706042    2576 apiserver.go:52] "Watching apiserver"
Jul 01 19:22:13.724963 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.724895    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 19:22:13.725429 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.725404    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-p6b5k","openshift-ovn-kubernetes/ovnkube-node-4ppkh","kube-system/global-pull-secret-syncer-2kwwj","kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal","openshift-image-registry/node-ca-2cg7l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal","openshift-multus/multus-additional-cni-plugins-8xg25","openshift-network-operator/iptables-alerter-hgqkp","kube-system/konnectivity-agent-pdkcn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8","openshift-cluster-node-tuning-operator/tuned-shkrb","openshift-dns/node-resolver-7ktbn","openshift-multus/multus-t2nzh","openshift-multus/network-metrics-daemon-6btbd"]
Jul 01 19:22:13.725644 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.725625    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 19:22:13.728214 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.728193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.730761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.730738    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 19:22:13.730864 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.730795    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 19:22:13.730864 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.730796    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 19:22:13.730864 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.730813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7sgsx\""
Jul 01 19:22:13.731591 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.731571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.733912 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.733887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-sys-fs\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.734037 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.733926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzpl\" (UniqueName: \"kubernetes.io/projected/3c886e46-aa74-453f-9b93-c790355e514f-kube-api-access-ztzpl\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.734037 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.733955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.734134 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.733984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-socket-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.734134 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.734087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-device-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.734134 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.734112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-registration-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.734270 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.734138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.734326 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.734264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.734379 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.734352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:13.734863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.734843    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 19:22:13.735072 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.735054    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 19:22:13.735144 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.735131    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 19:22:13.735421 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.735407    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 19:22:13.735565 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.735551    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fppgt\""
Jul 01 19:22:13.736729 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.736713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:13.736811 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.736793    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:13.739146 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.739124    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.741552 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.741533    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:13.742062 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.742045    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 19:22:13.744014 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.743989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.746506 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.746490    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.749248 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.749230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:13.749344 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.749301    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:13.750886 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.750870    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 19:22:13.752075 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.752060    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.752166 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.752124    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 19:22:13.752432 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.752410    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 19:22:13.752539 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.752410    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 19:22:13.752599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.752536    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 19:22:13.753479 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753456    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 19:22:13.753565 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753495    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 19:22:13.753565 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753508    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m7qcw\""
Jul 01 19:22:13.753671 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753515    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 19:22:13.753727 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753673    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9ptj\""
Jul 01 19:22:13.753727 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753685    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qsng4\""
Jul 01 19:22:13.753822 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753737    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 19:22:13.753822 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753767    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 19:22:13.754007 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.753992    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 19:22:13.754102 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.754088    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bs7rx\""
Jul 01 19:22:13.755055 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.755035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.755154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.755125    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 19:22:13.755434 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.755417    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:22:13.755513 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.755440    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 19:22:13.757888 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.757870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.764368 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764324    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h92cw\""
Jul 01 19:22:13.764475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764445    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 19:22:13.764475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764456    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g9s5d\""
Jul 01 19:22:13.764643 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764626    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:22:13.764719 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764703    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 19:22:13.764719 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764712    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:22:13.764828 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 19:22:13.764828 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.764731    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 19:22:13.773465 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.773434    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wqchf\""
Jul 01 19:22:13.775196 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.775157    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 19:17:13 +0000 UTC" deadline="2028-04-01 06:43:12.106963186 +0000 UTC"
Jul 01 19:22:13.775196 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.775195    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15347h20m58.331771629s"
Jul 01 19:22:13.806429 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.806402    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nmzns"
Jul 01 19:22:13.815733 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.815711    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nmzns"
Jul 01 19:22:13.827950 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.827926    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 19:22:13.834690 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-device-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.834827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a87f684-4294-4618-ba06-c37040b80606-kubelet-config\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.834827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovnkube-script-lib\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.834827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jzv\" (UniqueName: \"kubernetes.io/projected/256697b6-5404-4d73-94a9-d2363e8c37f1-kube-api-access-t4jzv\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.834827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysconfig\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.834827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysctl-d\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.834827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-device-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.834827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-lib-modules\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-registration-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35de78d3-1e02-4304-a370-96a53c6a794c-etc-tuned\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-iptables-alerter-script\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834913    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-registration-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-os-release\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-cni-multus\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.834979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-log-socket\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-system-cni-dir\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-var-lib-kubelet\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-conf-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.835145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-sys-fs\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835161    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-sys-fs\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3023be20-3d3a-4833-914f-1f98ea4ba5ee-tmp-dir\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-system-cni-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-slash\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxbrz\" (UniqueName: \"kubernetes.io/projected/3023be20-3d3a-4833-914f-1f98ea4ba5ee-kube-api-access-vxbrz\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-host\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35de78d3-1e02-4304-a370-96a53c6a794c-tmp\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b755486f-bcce-43d8-b6d7-c06e780555fe-host\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-host-slash\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-cnibin\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-etc-kubernetes\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-socket-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a87f684-4294-4618-ba06-c37040b80606-dbus\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-node-log\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.835599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-env-overrides\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-hostroot\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-multus-certs\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835628    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-socket-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-cni-netd\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovnkube-config\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-cnibin\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-os-release\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835794    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-sys\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlx58\" (UniqueName: \"kubernetes.io/projected/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-kube-api-access-jlx58\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/247b7370-a4e2-4213-a816-91ca08794aae-multus-daemon-config\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669lz\" (UniqueName: \"kubernetes.io/projected/247b7370-a4e2-4213-a816-91ca08794aae-kube-api-access-669lz\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-modprobe-d\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66r7h\" (UniqueName: \"kubernetes.io/projected/35de78d3-1e02-4304-a370-96a53c6a794c-kube-api-access-66r7h\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.836353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-cni-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.835996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-ovn\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-cni-bin\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-run\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3023be20-3d3a-4833-914f-1f98ea4ba5ee-hosts-file\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5x2k\" (UniqueName: \"kubernetes.io/projected/59d8dc7f-ebf5-41cb-8951-339301976b04-kube-api-access-j5x2k\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836266    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836285    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b2bb51a-496e-4929-b481-922ce49e4767-konnectivity-ca\") pod \"konnectivity-agent-pdkcn\" (UID: \"4b2bb51a-496e-4929-b481-922ce49e4767\") " pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-cni-bin\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-kubelet\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-etc-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jmx\" (UniqueName: \"kubernetes.io/projected/bc73ea4b-d504-4d79-be60-6eedf337dc0a-kube-api-access-b9jmx\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/247b7370-a4e2-4213-a816-91ca08794aae-cni-binary-copy\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.837100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovn-node-metrics-cert\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b755486f-bcce-43d8-b6d7-c06e780555fe-serviceca\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b2bb51a-496e-4929-b481-922ce49e4767-agent-certs\") pod \"konnectivity-agent-pdkcn\" (UID: \"4b2bb51a-496e-4929-b481-922ce49e4767\") " pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysctl-conf\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzpl\" (UniqueName: \"kubernetes.io/projected/3c886e46-aa74-453f-9b93-c790355e514f-kube-api-access-ztzpl\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-systemd\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-k8s-cni-cncf-io\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-netns\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-systemd\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-socket-dir-parent\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.836988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-kubernetes\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-kubelet\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-systemd-units\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.837933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-run-netns\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.838651 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-var-lib-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.838651 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c886e46-aa74-453f-9b93-c790355e514f-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.838651 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.838651 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.837214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bxgx\" (UniqueName: \"kubernetes.io/projected/b755486f-bcce-43d8-b6d7-c06e780555fe-kube-api-access-5bxgx\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.847908 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.847881    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 19:22:13.851373 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.851350    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzpl\" (UniqueName: \"kubernetes.io/projected/3c886e46-aa74-453f-9b93-c790355e514f-kube-api-access-ztzpl\") pod \"aws-ebs-csi-driver-node-m9cr8\" (UID: \"3c886e46-aa74-453f-9b93-c790355e514f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:13.912697 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.912665    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:13.926559 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:13.926507    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e19a46725c74590d2419ed8e9a32c3d.slice/crio-08fca976b34b1d92524554e805eb2c5ca7b4fa9245147bad34646bf94a0ea040 WatchSource:0}: Error finding container 08fca976b34b1d92524554e805eb2c5ca7b4fa9245147bad34646bf94a0ea040: Status 404 returned error can't find the container with id 08fca976b34b1d92524554e805eb2c5ca7b4fa9245147bad34646bf94a0ea040
Jul 01 19:22:13.926782 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:13.926762    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef75bb6a2b1f0b788d8d4b4eec583ca4.slice/crio-b3283fd48304f146bbeecd19803bb2647090a9fe4934aadfeb2fe56d3191e32d WatchSource:0}: Error finding container b3283fd48304f146bbeecd19803bb2647090a9fe4934aadfeb2fe56d3191e32d: Status 404 returned error can't find the container with id b3283fd48304f146bbeecd19803bb2647090a9fe4934aadfeb2fe56d3191e32d
Jul 01 19:22:13.933340 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.931062    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:22:13.938446 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a87f684-4294-4618-ba06-c37040b80606-kubelet-config\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.938554 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovnkube-script-lib\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.938554 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jzv\" (UniqueName: \"kubernetes.io/projected/256697b6-5404-4d73-94a9-d2363e8c37f1-kube-api-access-t4jzv\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.938554 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysconfig\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.938554 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysctl-d\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938561    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a87f684-4294-4618-ba06-c37040b80606-kubelet-config\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysconfig\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-lib-modules\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35de78d3-1e02-4304-a370-96a53c6a794c-etc-tuned\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938689    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysctl-d\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-iptables-alerter-script\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-os-release\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.938761 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-cni-multus\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938776    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-cni-multus\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-log-socket\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-os-release\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938862    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-log-socket\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-lib-modules\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.938963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-system-cni-dir\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-system-cni-dir\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-var-lib-kubelet\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-conf-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-conf-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939139    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-var-lib-kubelet\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.939170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3023be20-3d3a-4833-914f-1f98ea4ba5ee-tmp-dir\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939182    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovnkube-script-lib\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-system-cni-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-system-cni-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-slash\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxbrz\" (UniqueName: \"kubernetes.io/projected/3023be20-3d3a-4833-914f-1f98ea4ba5ee-kube-api-access-vxbrz\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939326    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-iptables-alerter-script\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-host\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939369    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-slash\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35de78d3-1e02-4304-a370-96a53c6a794c-tmp\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939452    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3023be20-3d3a-4833-914f-1f98ea4ba5ee-tmp-dir\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939484    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b755486f-bcce-43d8-b6d7-c06e780555fe-host\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b755486f-bcce-43d8-b6d7-c06e780555fe-host\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-host-slash\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.939580    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-cnibin\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.939866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-etc-kubernetes\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.939659    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:14.439634954 +0000 UTC m=+2.052678341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-host-slash\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939692    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-etc-kubernetes\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939706    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-cnibin\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a87f684-4294-4618-ba06-c37040b80606-dbus\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-node-log\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-env-overrides\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939817    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-node-log\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-hostroot\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939870    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a87f684-4294-4618-ba06-c37040b80606-dbus\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939993    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-host\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-multus-certs\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939944    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-hostroot\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.939892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-multus-certs\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-cni-netd\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.940863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovnkube-config\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-cnibin\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940238    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-cni-netd\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-cnibin\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-env-overrides\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-os-release\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-sys\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlx58\" (UniqueName: \"kubernetes.io/projected/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-kube-api-access-jlx58\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/247b7370-a4e2-4213-a816-91ca08794aae-multus-daemon-config\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669lz\" (UniqueName: \"kubernetes.io/projected/247b7370-a4e2-4213-a816-91ca08794aae-kube-api-access-669lz\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940436    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-os-release\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-modprobe-d\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66r7h\" (UniqueName: \"kubernetes.io/projected/35de78d3-1e02-4304-a370-96a53c6a794c-kube-api-access-66r7h\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-cni-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940603    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-cni-dir\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.941738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.940991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-ovn\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941007    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-cni-bin\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941094    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/247b7370-a4e2-4213-a816-91ca08794aae-multus-daemon-config\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-cni-bin\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/256697b6-5404-4d73-94a9-d2363e8c37f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-run\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941149    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-modprobe-d\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941178    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-sys\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-run\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3023be20-3d3a-4833-914f-1f98ea4ba5ee-hosts-file\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941221    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-ovn\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/256697b6-5404-4d73-94a9-d2363e8c37f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5x2k\" (UniqueName: \"kubernetes.io/projected/59d8dc7f-ebf5-41cb-8951-339301976b04-kube-api-access-j5x2k\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941311    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3023be20-3d3a-4833-914f-1f98ea4ba5ee-hosts-file\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b2bb51a-496e-4929-b481-922ce49e4767-konnectivity-ca\") pod \"konnectivity-agent-pdkcn\" (UID: \"4b2bb51a-496e-4929-b481-922ce49e4767\") " pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:13.942698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovnkube-config\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-cni-bin\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-kubelet\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-cni-bin\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-var-lib-kubelet\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-etc-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jmx\" (UniqueName: \"kubernetes.io/projected/bc73ea4b-d504-4d79-be60-6eedf337dc0a-kube-api-access-b9jmx\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.941541    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/247b7370-a4e2-4213-a816-91ca08794aae-cni-binary-copy\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35de78d3-1e02-4304-a370-96a53c6a794c-etc-tuned\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovn-node-metrics-cert\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.941603    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret podName:3a87f684-4294-4618-ba06-c37040b80606 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:14.441590623 +0000 UTC m=+2.054633992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret") pod "global-pull-secret-syncer-2kwwj" (UID: "3a87f684-4294-4618-ba06-c37040b80606") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941541    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-etc-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b755486f-bcce-43d8-b6d7-c06e780555fe-serviceca\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b2bb51a-496e-4929-b481-922ce49e4767-agent-certs\") pod \"konnectivity-agent-pdkcn\" (UID: \"4b2bb51a-496e-4929-b481-922ce49e4767\") " pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:13.943616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysctl-conf\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-systemd\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-k8s-cni-cncf-io\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-netns\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35de78d3-1e02-4304-a370-96a53c6a794c-tmp\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-systemd\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941979    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-sysctl-conf\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.941988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b2bb51a-496e-4929-b481-922ce49e4767-konnectivity-ca\") pod \"konnectivity-agent-pdkcn\" (UID: \"4b2bb51a-496e-4929-b481-922ce49e4767\") " pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-k8s-cni-cncf-io\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-systemd\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-host-run-netns\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-systemd\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-socket-dir-parent\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b755486f-bcce-43d8-b6d7-c06e780555fe-serviceca\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-kubernetes\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-kubelet\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944378 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942191    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/247b7370-a4e2-4213-a816-91ca08794aae-cni-binary-copy\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/247b7370-a4e2-4213-a816-91ca08794aae-multus-socket-dir-parent\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-systemd-units\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-run-netns\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35de78d3-1e02-4304-a370-96a53c6a794c-etc-kubernetes\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942271    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-kubelet\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942291    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-systemd-units\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-var-lib-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942348    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-host-run-netns\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-var-lib-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bxgx\" (UniqueName: \"kubernetes.io/projected/b755486f-bcce-43d8-b6d7-c06e780555fe-kube-api-access-5bxgx\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.942429    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc73ea4b-d504-4d79-be60-6eedf337dc0a-run-openvswitch\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.944208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b2bb51a-496e-4929-b481-922ce49e4767-agent-certs\") pod \"konnectivity-agent-pdkcn\" (UID: \"4b2bb51a-496e-4929-b481-922ce49e4767\") " pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:13.944929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.944304    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc73ea4b-d504-4d79-be60-6eedf337dc0a-ovn-node-metrics-cert\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.951338 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.951310    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jzv\" (UniqueName: \"kubernetes.io/projected/256697b6-5404-4d73-94a9-d2363e8c37f1-kube-api-access-t4jzv\") pod \"multus-additional-cni-plugins-8xg25\" (UID: \"256697b6-5404-4d73-94a9-d2363e8c37f1\") " pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:13.952857 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.952827    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlx58\" (UniqueName: \"kubernetes.io/projected/af3076a6-12bd-405c-8ec3-d1fce98fdb9a-kube-api-access-jlx58\") pod \"iptables-alerter-hgqkp\" (UID: \"af3076a6-12bd-405c-8ec3-d1fce98fdb9a\") " pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:13.953061 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.952974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bxgx\" (UniqueName: \"kubernetes.io/projected/b755486f-bcce-43d8-b6d7-c06e780555fe-kube-api-access-5bxgx\") pod \"node-ca-2cg7l\" (UID: \"b755486f-bcce-43d8-b6d7-c06e780555fe\") " pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:13.953199 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.953168    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:13.953199 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.953196    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:13.953286 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.953211    2576 projected.go:196] Error preparing data for projected volume kube-api-access-4zv4h for pod openshift-network-diagnostics/network-check-target-p6b5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:13.953286 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.953251    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxbrz\" (UniqueName: \"kubernetes.io/projected/3023be20-3d3a-4833-914f-1f98ea4ba5ee-kube-api-access-vxbrz\") pod \"node-resolver-7ktbn\" (UID: \"3023be20-3d3a-4833-914f-1f98ea4ba5ee\") " pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:13.953286 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.953260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5x2k\" (UniqueName: \"kubernetes.io/projected/59d8dc7f-ebf5-41cb-8951-339301976b04-kube-api-access-j5x2k\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:13.953286 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:13.953278    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h podName:60335b20-da5c-44f7-8ef1-e1c9858fbff9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:14.453258248 +0000 UTC m=+2.066301631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4zv4h" (UniqueName: "kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h") pod "network-check-target-p6b5k" (UID: "60335b20-da5c-44f7-8ef1-e1c9858fbff9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:13.954527 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.954501    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-669lz\" (UniqueName: \"kubernetes.io/projected/247b7370-a4e2-4213-a816-91ca08794aae-kube-api-access-669lz\") pod \"multus-t2nzh\" (UID: \"247b7370-a4e2-4213-a816-91ca08794aae\") " pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:13.954910 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.954889    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66r7h\" (UniqueName: \"kubernetes.io/projected/35de78d3-1e02-4304-a370-96a53c6a794c-kube-api-access-66r7h\") pod \"tuned-shkrb\" (UID: \"35de78d3-1e02-4304-a370-96a53c6a794c\") " pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:13.955562 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.955545    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jmx\" (UniqueName: \"kubernetes.io/projected/bc73ea4b-d504-4d79-be60-6eedf337dc0a-kube-api-access-b9jmx\") pod \"ovnkube-node-4ppkh\" (UID: \"bc73ea4b-d504-4d79-be60-6eedf337dc0a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:13.980070 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:13.980044    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:14.060252 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.060149    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8"
Jul 01 19:22:14.064494 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.064463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2nzh"
Jul 01 19:22:14.072122 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.072093    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c886e46_aa74_453f_9b93_c790355e514f.slice/crio-f4345e970ca2b31218404c306837196bd8215e291835a5ef496addb70c41f19d WatchSource:0}: Error finding container f4345e970ca2b31218404c306837196bd8215e291835a5ef496addb70c41f19d: Status 404 returned error can't find the container with id f4345e970ca2b31218404c306837196bd8215e291835a5ef496addb70c41f19d
Jul 01 19:22:14.074860 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.074827    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247b7370_a4e2_4213_a816_91ca08794aae.slice/crio-182c7b2e05a9f340f5738e6992bc7df221788357f97070caea0adf2b7e3ae4f7 WatchSource:0}: Error finding container 182c7b2e05a9f340f5738e6992bc7df221788357f97070caea0adf2b7e3ae4f7: Status 404 returned error can't find the container with id 182c7b2e05a9f340f5738e6992bc7df221788357f97070caea0adf2b7e3ae4f7
Jul 01 19:22:14.079609 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.079575    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cg7l"
Jul 01 19:22:14.089257 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.089225    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb755486f_bcce_43d8_b6d7_c06e780555fe.slice/crio-a5891dbcd8c17891b282b7e37d6636d540c281337080d089518a5107c422e6a5 WatchSource:0}: Error finding container a5891dbcd8c17891b282b7e37d6636d540c281337080d089518a5107c422e6a5: Status 404 returned error can't find the container with id a5891dbcd8c17891b282b7e37d6636d540c281337080d089518a5107c422e6a5
Jul 01 19:22:14.098435 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.098416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:14.107518 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.107492    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2bb51a_496e_4929_b481_922ce49e4767.slice/crio-ea2a6e2be4f777e7ee954b5c87fc933754cee58150b7de7134a7385b7b001311 WatchSource:0}: Error finding container ea2a6e2be4f777e7ee954b5c87fc933754cee58150b7de7134a7385b7b001311: Status 404 returned error can't find the container with id ea2a6e2be4f777e7ee954b5c87fc933754cee58150b7de7134a7385b7b001311
Jul 01 19:22:14.119759 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.119740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xg25"
Jul 01 19:22:14.129493 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.129468    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256697b6_5404_4d73_94a9_d2363e8c37f1.slice/crio-e32e95cde21fc0522001ee5d2fa80ecbe0d2a07fc21a1d50e09c788778e1e9da WatchSource:0}: Error finding container e32e95cde21fc0522001ee5d2fa80ecbe0d2a07fc21a1d50e09c788778e1e9da: Status 404 returned error can't find the container with id e32e95cde21fc0522001ee5d2fa80ecbe0d2a07fc21a1d50e09c788778e1e9da
Jul 01 19:22:14.135027 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.135004    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:14.144320 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.144298    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc73ea4b_d504_4d79_be60_6eedf337dc0a.slice/crio-dc36740c40cb2a4c1c670fa6072fd5a6f11ce949574849d2ec32e9e6fab6a873 WatchSource:0}: Error finding container dc36740c40cb2a4c1c670fa6072fd5a6f11ce949574849d2ec32e9e6fab6a873: Status 404 returned error can't find the container with id dc36740c40cb2a4c1c670fa6072fd5a6f11ce949574849d2ec32e9e6fab6a873
Jul 01 19:22:14.147880 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.147864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shkrb"
Jul 01 19:22:14.156544 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.156522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35de78d3_1e02_4304_a370_96a53c6a794c.slice/crio-f030beaab772ad4fd5cd920944218d9c85bb97dee6a37f69d86ff46728e7071f WatchSource:0}: Error finding container f030beaab772ad4fd5cd920944218d9c85bb97dee6a37f69d86ff46728e7071f: Status 404 returned error can't find the container with id f030beaab772ad4fd5cd920944218d9c85bb97dee6a37f69d86ff46728e7071f
Jul 01 19:22:14.156691 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.156551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7ktbn"
Jul 01 19:22:14.161968 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.161949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hgqkp"
Jul 01 19:22:14.165705 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.165681    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3023be20_3d3a_4833_914f_1f98ea4ba5ee.slice/crio-85c88d8f3a7d5fd72a0e9d47eec7ef8a63a7914df815ea3943485b01ff901e2d WatchSource:0}: Error finding container 85c88d8f3a7d5fd72a0e9d47eec7ef8a63a7914df815ea3943485b01ff901e2d: Status 404 returned error can't find the container with id 85c88d8f3a7d5fd72a0e9d47eec7ef8a63a7914df815ea3943485b01ff901e2d
Jul 01 19:22:14.170652 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:14.170629    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3076a6_12bd_405c_8ec3_d1fce98fdb9a.slice/crio-673c91cbcd94f8b09c07ff48830fd865aee90d1d7ec9c0b422b1af151dfab732 WatchSource:0}: Error finding container 673c91cbcd94f8b09c07ff48830fd865aee90d1d7ec9c0b422b1af151dfab732: Status 404 returned error can't find the container with id 673c91cbcd94f8b09c07ff48830fd865aee90d1d7ec9c0b422b1af151dfab732
Jul 01 19:22:14.446793 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.446681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:14.446793 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.446775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:14.447044 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.446869    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:14.447044 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.446957    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:14.447044 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.446970    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:15.446951693 +0000 UTC m=+3.059995077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:14.447044 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.447037    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret podName:3a87f684-4294-4618-ba06-c37040b80606 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:15.447017962 +0000 UTC m=+3.060061334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret") pod "global-pull-secret-syncer-2kwwj" (UID: "3a87f684-4294-4618-ba06-c37040b80606") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:14.547235 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.547199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:14.547431 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.547416    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:14.547504 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.547483    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:14.547504 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.547495    2576 projected.go:196] Error preparing data for projected volume kube-api-access-4zv4h for pod openshift-network-diagnostics/network-check-target-p6b5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:14.547610 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.547580    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h podName:60335b20-da5c-44f7-8ef1-e1c9858fbff9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:15.54755852 +0000 UTC m=+3.160601905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zv4h" (UniqueName: "kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h") pod "network-check-target-p6b5k" (UID: "60335b20-da5c-44f7-8ef1-e1c9858fbff9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:14.550080 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.550052    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:14.801835 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.799163    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:14.801835 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:14.799372    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:14.819268 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.819174    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:17:14 +0000 UTC" deadline="2028-04-16 13:41:55.693326204 +0000 UTC"
Jul 01 19:22:14.819268 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.819236    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15714h19m40.874095673s"
Jul 01 19:22:14.827564 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.827497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2nzh" event={"ID":"247b7370-a4e2-4213-a816-91ca08794aae","Type":"ContainerStarted","Data":"182c7b2e05a9f340f5738e6992bc7df221788357f97070caea0adf2b7e3ae4f7"}
Jul 01 19:22:14.838570 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.838532    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8" event={"ID":"3c886e46-aa74-453f-9b93-c790355e514f","Type":"ContainerStarted","Data":"f4345e970ca2b31218404c306837196bd8215e291835a5ef496addb70c41f19d"}
Jul 01 19:22:14.847174 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.847135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shkrb" event={"ID":"35de78d3-1e02-4304-a370-96a53c6a794c","Type":"ContainerStarted","Data":"f030beaab772ad4fd5cd920944218d9c85bb97dee6a37f69d86ff46728e7071f"}
Jul 01 19:22:14.861504 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.861411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal" event={"ID":"ef75bb6a2b1f0b788d8d4b4eec583ca4","Type":"ContainerStarted","Data":"b3283fd48304f146bbeecd19803bb2647090a9fe4934aadfeb2fe56d3191e32d"}
Jul 01 19:22:14.874472 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.874429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal" event={"ID":"7e19a46725c74590d2419ed8e9a32c3d","Type":"ContainerStarted","Data":"08fca976b34b1d92524554e805eb2c5ca7b4fa9245147bad34646bf94a0ea040"}
Jul 01 19:22:14.890411 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.890330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hgqkp" event={"ID":"af3076a6-12bd-405c-8ec3-d1fce98fdb9a","Type":"ContainerStarted","Data":"673c91cbcd94f8b09c07ff48830fd865aee90d1d7ec9c0b422b1af151dfab732"}
Jul 01 19:22:14.903280 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.903240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7ktbn" event={"ID":"3023be20-3d3a-4833-914f-1f98ea4ba5ee","Type":"ContainerStarted","Data":"85c88d8f3a7d5fd72a0e9d47eec7ef8a63a7914df815ea3943485b01ff901e2d"}
Jul 01 19:22:14.916949 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.916911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"dc36740c40cb2a4c1c670fa6072fd5a6f11ce949574849d2ec32e9e6fab6a873"}
Jul 01 19:22:14.935577 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.935535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerStarted","Data":"e32e95cde21fc0522001ee5d2fa80ecbe0d2a07fc21a1d50e09c788778e1e9da"}
Jul 01 19:22:14.947424 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.947383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pdkcn" event={"ID":"4b2bb51a-496e-4929-b481-922ce49e4767","Type":"ContainerStarted","Data":"ea2a6e2be4f777e7ee954b5c87fc933754cee58150b7de7134a7385b7b001311"}
Jul 01 19:22:14.957428 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:14.957389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cg7l" event={"ID":"b755486f-bcce-43d8-b6d7-c06e780555fe","Type":"ContainerStarted","Data":"a5891dbcd8c17891b282b7e37d6636d540c281337080d089518a5107c422e6a5"}
Jul 01 19:22:15.454811 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:15.454751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:15.455027 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:15.454846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:15.455027 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.455010    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:15.455163 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.455079    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret podName:3a87f684-4294-4618-ba06-c37040b80606 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:17.455059869 +0000 UTC m=+5.068103255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret") pod "global-pull-secret-syncer-2kwwj" (UID: "3a87f684-4294-4618-ba06-c37040b80606") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:15.455526 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.455506    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:15.455616 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.455567    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:17.455549069 +0000 UTC m=+5.068592450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:15.555590 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:15.555483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:15.555788 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.555689    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:15.555788 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.555715    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:15.555788 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.555728    2576 projected.go:196] Error preparing data for projected volume kube-api-access-4zv4h for pod openshift-network-diagnostics/network-check-target-p6b5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:15.555788 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.555789    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h podName:60335b20-da5c-44f7-8ef1-e1c9858fbff9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:17.555770313 +0000 UTC m=+5.168813695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zv4h" (UniqueName: "kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h") pod "network-check-target-p6b5k" (UID: "60335b20-da5c-44f7-8ef1-e1c9858fbff9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:15.790286 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:15.789599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:15.790286 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:15.789634    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:15.790286 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.789747    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:15.790286 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:15.789899    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:15.820047 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:15.819993    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:17:14 +0000 UTC" deadline="2028-03-02 11:49:55.12251945 +0000 UTC"
Jul 01 19:22:15.820047 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:15.820043    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14632h27m39.302481653s"
Jul 01 19:22:16.602339 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:16.602304    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:16.791010 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:16.790372    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:16.791010 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:16.790521    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:17.469860 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:17.469790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:17.470443 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:17.469868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:17.470443 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.470035    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:17.470443 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.470107    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret podName:3a87f684-4294-4618-ba06-c37040b80606 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:21.470083806 +0000 UTC m=+9.083127182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret") pod "global-pull-secret-syncer-2kwwj" (UID: "3a87f684-4294-4618-ba06-c37040b80606") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:17.470443 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.470181    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:17.470443 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.470217    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:21.470204663 +0000 UTC m=+9.083248036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:17.570855 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:17.570731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:17.571088 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.570998    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:17.571088 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.571031    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:17.571088 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.571045    2576 projected.go:196] Error preparing data for projected volume kube-api-access-4zv4h for pod openshift-network-diagnostics/network-check-target-p6b5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:17.571292 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.571114    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h podName:60335b20-da5c-44f7-8ef1-e1c9858fbff9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:21.571094122 +0000 UTC m=+9.184137511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zv4h" (UniqueName: "kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h") pod "network-check-target-p6b5k" (UID: "60335b20-da5c-44f7-8ef1-e1c9858fbff9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:17.790091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:17.790001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:17.790303 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.790135    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:17.790603 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:17.790571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:17.790703 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:17.790684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:18.790162 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:18.790129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:18.790603 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:18.790277    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:19.789675 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:19.789519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:19.789675 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:19.789664    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:19.789955 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:19.789531    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:19.790400 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:19.789947    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:20.789933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:20.789682    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:20.790084 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:20.790035    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:21.506587 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:21.506545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:21.507099 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:21.506612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:21.507099 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.506721    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:21.507099 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.506803    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:29.506780384 +0000 UTC m=+17.119823749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:21.507099 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.506726    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:21.507099 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.506871    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret podName:3a87f684-4294-4618-ba06-c37040b80606 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:29.506853851 +0000 UTC m=+17.119897220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret") pod "global-pull-secret-syncer-2kwwj" (UID: "3a87f684-4294-4618-ba06-c37040b80606") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:21.608097 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:21.608011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:21.608283 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.608180    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:21.608283 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.608209    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:21.608283 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.608224    2576 projected.go:196] Error preparing data for projected volume kube-api-access-4zv4h for pod openshift-network-diagnostics/network-check-target-p6b5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:21.608442 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.608292    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h podName:60335b20-da5c-44f7-8ef1-e1c9858fbff9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:29.608272669 +0000 UTC m=+17.221316040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zv4h" (UniqueName: "kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h") pod "network-check-target-p6b5k" (UID: "60335b20-da5c-44f7-8ef1-e1c9858fbff9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:21.790221 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:21.790145    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:21.790382 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:21.790153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:21.790382 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.790280    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:21.790382 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:21.790365    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:22.790850 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:22.790816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:22.793501 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:22.793463    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:23.789427 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:23.789391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:23.789626 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:23.789602    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:23.789626 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:23.789613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:23.789774 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:23.789752    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:24.789226 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:24.789197    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:24.789613 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:24.789303    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:25.790097 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:25.790057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:25.790481 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:25.790056    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:25.790481 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:25.790180    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:25.790481 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:25.790225    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:26.789461 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:26.789427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:26.789647 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:26.789555    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:27.789097 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:27.789067    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:27.789565 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:27.789067    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:27.789565 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:27.789199    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:27.789565 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:27.789277    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:28.789849 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:28.789816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:28.790288 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:28.789964    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:29.571086 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:29.571051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:29.571319 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:29.571136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:29.571319 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.571225    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:29.571319 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.571287    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:29.571319 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.571309    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:45.571287616 +0000 UTC m=+33.184330996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:29.571492 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.571338    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret podName:3a87f684-4294-4618-ba06-c37040b80606 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:45.571322783 +0000 UTC m=+33.184366166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret") pod "global-pull-secret-syncer-2kwwj" (UID: "3a87f684-4294-4618-ba06-c37040b80606") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:29.672274 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:29.672238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:29.672468 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.672426    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:29.672468 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.672460    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:29.672555 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.672472    2576 projected.go:196] Error preparing data for projected volume kube-api-access-4zv4h for pod openshift-network-diagnostics/network-check-target-p6b5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:29.672555 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.672539    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h podName:60335b20-da5c-44f7-8ef1-e1c9858fbff9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:45.672519481 +0000 UTC m=+33.285562846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zv4h" (UniqueName: "kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h") pod "network-check-target-p6b5k" (UID: "60335b20-da5c-44f7-8ef1-e1c9858fbff9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:29.789515 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:29.789477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:29.789715 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.789609    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:29.789715 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:29.789663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:29.789820 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:29.789757    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:30.789503 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:30.789468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:30.790015 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:30.789605    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:31.789748 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:31.789714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:31.790186 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:31.789819    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:31.790186 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:31.789870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:31.790186 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:31.789982    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:32.790721 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:32.790681    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:32.791213 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:32.790806    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:33.789981 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:33.789944    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:33.790165 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:33.790081    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:33.790165 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:33.790130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:33.790311 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:33.790251    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:34.790057 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:34.790019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:34.790413 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:34.790157    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:35.790293 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:35.789999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:35.790960 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:35.789999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:35.790960 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:35.790389    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:35.790960 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:35.790437    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:36.019732 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.019695    2576 generic.go:356] "Generic (PLEG): container finished" podID="256697b6-5404-4d73-94a9-d2363e8c37f1" containerID="092a33d968f0a8207ed6a63e5aa0ca9b1808a5f5d7d15ed043b09b1e4afcc960" exitCode=0
Jul 01 19:22:36.019906 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.019777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerDied","Data":"092a33d968f0a8207ed6a63e5aa0ca9b1808a5f5d7d15ed043b09b1e4afcc960"}
Jul 01 19:22:36.021267 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.021225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pdkcn" event={"ID":"4b2bb51a-496e-4929-b481-922ce49e4767","Type":"ContainerStarted","Data":"1f83a9c28cafe3081b4658710a360244073ccc792c37c9c745c333564a466a91"}
Jul 01 19:22:36.022566 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.022537    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cg7l" event={"ID":"b755486f-bcce-43d8-b6d7-c06e780555fe","Type":"ContainerStarted","Data":"a5b217452ac4f2558e33bda1327b2efab2c5dba3b15c6f7756c8e0d8a25d0e56"}
Jul 01 19:22:36.024084 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.024063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2nzh" event={"ID":"247b7370-a4e2-4213-a816-91ca08794aae","Type":"ContainerStarted","Data":"19517f5f3f6fb397ddc0254460f17be13082f1fb785923e0aa31c5dc68b814d6"}
Jul 01 19:22:36.025359 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.025341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8" event={"ID":"3c886e46-aa74-453f-9b93-c790355e514f","Type":"ContainerStarted","Data":"14c390992e64952b4e038d83f1cad085e9c99ea9810fddcc7e76d6fe2da5dc70"}
Jul 01 19:22:36.026672 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.026651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shkrb" event={"ID":"35de78d3-1e02-4304-a370-96a53c6a794c","Type":"ContainerStarted","Data":"d4e96a4cab369d89a15fff5f8cc08c1cdedb6dc2f83ff9a29948a3efa143c06b"}
Jul 01 19:22:36.027968 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.027947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal" event={"ID":"ef75bb6a2b1f0b788d8d4b4eec583ca4","Type":"ContainerStarted","Data":"3884233bb6ac6243a3f20ed25a97f018fe9900fbe555f74f3b444864139cd2c1"}
Jul 01 19:22:36.029464 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.029445    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e19a46725c74590d2419ed8e9a32c3d" containerID="646e0ab6f02c087f565ed16e66ed9978b8c7109cdb19236b6e32fb35fec2fecd" exitCode=0
Jul 01 19:22:36.029540 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.029504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal" event={"ID":"7e19a46725c74590d2419ed8e9a32c3d","Type":"ContainerDied","Data":"646e0ab6f02c087f565ed16e66ed9978b8c7109cdb19236b6e32fb35fec2fecd"}
Jul 01 19:22:36.031049 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.031030    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7ktbn" event={"ID":"3023be20-3d3a-4833-914f-1f98ea4ba5ee","Type":"ContainerStarted","Data":"c68e54aa4f2467087ec4952d1e6e738d05c34324e140582147e66d78dc776b2c"}
Jul 01 19:22:36.034226 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.034205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"b7b6041225592751db5c18a57fa8c588e4b06e7cadc08d6a0111513777b0e726"}
Jul 01 19:22:36.034301 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.034268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"5be22f3790efed0c7f01c540802c6fc0b13cbe25a0adb5331ad5b669140183d9"}
Jul 01 19:22:36.034339 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.034313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"601f44f70dc6020c0d55d527a6e363ab3e23e63e35e0892bc8e07fee2f706968"}
Jul 01 19:22:36.034374 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.034355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"3dfff7e8979cb7de57a44f621014f8e6bbb992f3a0dce9a7184b0aaeec81745e"}
Jul 01 19:22:36.034406 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.034385    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"9abaaa267acc71f17b175114a762c26cb55e22cf9b9997923a7d2f06369dfabe"}
Jul 01 19:22:36.034441 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.034411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"ed25f189e6ab67d9260b43fb4b151bdf027b0c6cbbff5ff31759cb36270090a3"}
Jul 01 19:22:36.087482 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.087448    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:36.088027 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.087998    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:36.095020 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.094961    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-shkrb" podStartSLOduration=2.161265223 podStartE2EDuration="23.094941172s" totalImagesPullingTime="20.933675949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:36.077463853 +0000 UTC m=+23.690507241" watchObservedRunningTime="2026-07-01 19:22:36.094941172 +0000 UTC m=+23.707984560"
Jul 01 19:22:36.124460 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.124405    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t2nzh" podStartSLOduration=2.109798785 podStartE2EDuration="23.124390816s" totalImagesPullingTime="21.014592031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:36.123933588 +0000 UTC m=+23.736976971" watchObservedRunningTime="2026-07-01 19:22:36.124390816 +0000 UTC m=+23.737434252"
Jul 01 19:22:36.165975 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.165898    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pdkcn" podStartSLOduration=10.971759918 podStartE2EDuration="23.165877942s" totalImagesPullingTime="12.194118024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:36.143245408 +0000 UTC m=+23.756288795" watchObservedRunningTime="2026-07-01 19:22:36.165877942 +0000 UTC m=+23.778921329"
Jul 01 19:22:36.166238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.166211    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7ktbn" podStartSLOduration=2.307209428 podStartE2EDuration="23.16620009s" totalImagesPullingTime="20.858990662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:36.165521779 +0000 UTC m=+23.778565167" watchObservedRunningTime="2026-07-01 19:22:36.16620009 +0000 UTC m=+23.779243477"
Jul 01 19:22:36.182480 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.182419    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-50.ec2.internal" podStartSLOduration=22.182397505 podStartE2EDuration="22.182397505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:36.182033167 +0000 UTC m=+23.795076555" watchObservedRunningTime="2026-07-01 19:22:36.182397505 +0000 UTC m=+23.795440896"
Jul 01 19:22:36.209065 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.209010    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2cg7l" podStartSLOduration=2.273357406 podStartE2EDuration="23.208990787s" totalImagesPullingTime="20.935633381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:36.208563177 +0000 UTC m=+23.821606576" watchObservedRunningTime="2026-07-01 19:22:36.208990787 +0000 UTC m=+23.822034173"
Jul 01 19:22:36.743335 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.743309    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 19:22:36.789286 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.789252    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:36.789476 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:36.789379    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:36.814085 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.813926    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T19:22:36.743327534Z","UUID":"25d6075d-563e-4e07-bd3c-22b5b2e6d687","Handler":null,"Name":"","Endpoint":""}
Jul 01 19:22:36.815317 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.815298    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 19:22:36.815431 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:36.815335    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 19:22:37.039232 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.039179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8" event={"ID":"3c886e46-aa74-453f-9b93-c790355e514f","Type":"ContainerStarted","Data":"3c239c03535f9782479bae1bd26322a88a190993b09aa2e28137c3ef6aa3cd94"}
Jul 01 19:22:37.041108 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.041079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal" event={"ID":"7e19a46725c74590d2419ed8e9a32c3d","Type":"ContainerStarted","Data":"5e5ae8ae57ab108567fee88fdaf7fd8cda34666b27566b75732a95f0c46f006f"}
Jul 01 19:22:37.042708 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.042609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hgqkp" event={"ID":"af3076a6-12bd-405c-8ec3-d1fce98fdb9a","Type":"ContainerStarted","Data":"292996b0e6884550e6a600f8fc9d348647c8dbde3a1a0af53581ecf9a4540aa7"}
Jul 01 19:22:37.043675 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.043655    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:37.043795 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.043722    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pdkcn"
Jul 01 19:22:37.057869 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.057801    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-50.ec2.internal" podStartSLOduration=24.057780476 podStartE2EDuration="24.057780476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:37.057682206 +0000 UTC m=+24.670725656" watchObservedRunningTime="2026-07-01 19:22:37.057780476 +0000 UTC m=+24.670823891"
Jul 01 19:22:37.093837 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.093731    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hgqkp" podStartSLOduration=3.17380784 podStartE2EDuration="24.093716297s" totalImagesPullingTime="20.919908457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:37.093333303 +0000 UTC m=+24.706376692" watchObservedRunningTime="2026-07-01 19:22:37.093716297 +0000 UTC m=+24.706759691"
Jul 01 19:22:37.789824 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.789786    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:37.790143 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:37.789937    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:37.790143 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:37.789784    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:37.790262 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:37.790184    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:38.793399 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:38.793369    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:38.793810 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:38.793480    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:39.051695 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:39.051653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"b5ad091572bdb24fb349dbf63e276133c8116ca79ff9eadf279c790c529f714a"}
Jul 01 19:22:39.053944 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:39.053897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8" event={"ID":"3c886e46-aa74-453f-9b93-c790355e514f","Type":"ContainerStarted","Data":"0ec2851fdb4c3ba1ea0d5a9f314a37a94e0166a8961b91301b5f7188d41691a2"}
Jul 01 19:22:39.075019 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:39.074968    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cr8" podStartSLOduration=2.268794905 podStartE2EDuration="26.07494964s" totalImagesPullingTime="23.806154735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:39.074463917 +0000 UTC m=+26.687507325" watchObservedRunningTime="2026-07-01 19:22:39.07494964 +0000 UTC m=+26.687993021"
Jul 01 19:22:39.789252 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:39.789219    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:39.789454 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:39.789350    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:39.789454 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:39.789409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:39.789582 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:39.789531    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:40.790034 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:40.789819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:40.790912 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:40.790148    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:41.061703 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.061666    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" event={"ID":"bc73ea4b-d504-4d79-be60-6eedf337dc0a","Type":"ContainerStarted","Data":"004631a50c3853b7fecf6ca170a37c176a1f20244389b67e8456d78884af1899"}
Jul 01 19:22:41.062001 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.061984    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:41.063525 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.063496    2576 generic.go:356] "Generic (PLEG): container finished" podID="256697b6-5404-4d73-94a9-d2363e8c37f1" containerID="60caf2d5dd427fffc6cb341e1d9c3de2c2313a0eb04a6423234938a94dcf26e8" exitCode=0
Jul 01 19:22:41.063612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.063550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerDied","Data":"60caf2d5dd427fffc6cb341e1d9c3de2c2313a0eb04a6423234938a94dcf26e8"}
Jul 01 19:22:41.078465 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.078439    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:41.093880 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.093821    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh" podStartSLOduration=7.041913647 podStartE2EDuration="28.093807346s" totalImagesPullingTime="21.051893699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:41.093530835 +0000 UTC m=+28.706574222" watchObservedRunningTime="2026-07-01 19:22:41.093807346 +0000 UTC m=+28.706850748"
Jul 01 19:22:41.789467 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.789290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:41.789694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:41.789291    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:41.789694 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:41.789535    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:41.789694 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:41.789613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:42.066154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:42.066054    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:42.066154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:42.066130    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:42.081083 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:42.081048    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:22:42.791017 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:42.790893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:42.791163 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:42.791002    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:42.829334 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:42.829302    2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256697b6_5404_4d73_94a9_d2363e8c37f1.slice/crio-conmon-f53c9bcb3e55bc7f8f9cf58575b474c65d42fbead7b4f59ff70ef58e6ce2edd3.scope\": RecentStats: unable to find data in memory cache]"
Jul 01 19:22:43.069960 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.069911    2576 generic.go:356] "Generic (PLEG): container finished" podID="256697b6-5404-4d73-94a9-d2363e8c37f1" containerID="f53c9bcb3e55bc7f8f9cf58575b474c65d42fbead7b4f59ff70ef58e6ce2edd3" exitCode=0
Jul 01 19:22:43.070406 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.070013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerDied","Data":"f53c9bcb3e55bc7f8f9cf58575b474c65d42fbead7b4f59ff70ef58e6ce2edd3"}
Jul 01 19:22:43.149251 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.149222    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p6b5k"]
Jul 01 19:22:43.149454 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.149333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:43.149454 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:43.149412    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:43.152825 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.152804    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kwwj"]
Jul 01 19:22:43.152933 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.152885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:43.153001 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:43.152982    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:43.154196 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.154168    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6btbd"]
Jul 01 19:22:43.154321 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:43.154272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:43.154396 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:43.154374    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:44.790091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:44.790048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:44.790091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:44.790083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:44.790528 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:44.790048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:44.790528 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:44.790174    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:44.790528 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:44.790344    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:44.790528 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:44.790411    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:45.077170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:45.077092    2576 generic.go:356] "Generic (PLEG): container finished" podID="256697b6-5404-4d73-94a9-d2363e8c37f1" containerID="bde8e2b82e6bcbe0e5e341e2d2fe49c3b5b3a7c54f4153b4097086406905e46b" exitCode=0
Jul 01 19:22:45.077170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:45.077157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerDied","Data":"bde8e2b82e6bcbe0e5e341e2d2fe49c3b5b3a7c54f4153b4097086406905e46b"}
Jul 01 19:22:45.583902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:45.583865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:45.584133 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:45.583941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:45.584133 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.584040    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:45.584133 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.584038    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:45.584133 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.584093    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret podName:3a87f684-4294-4618-ba06-c37040b80606 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:17.584078104 +0000 UTC m=+65.197121487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret") pod "global-pull-secret-syncer-2kwwj" (UID: "3a87f684-4294-4618-ba06-c37040b80606") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:45.584133 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.584112    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:17.584102602 +0000 UTC m=+65.197145972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:45.685047 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:45.685013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:45.685245 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.685192    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:45.685245 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.685217    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:45.685245 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.685231    2576 projected.go:196] Error preparing data for projected volume kube-api-access-4zv4h for pod openshift-network-diagnostics/network-check-target-p6b5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:45.685383 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:45.685291    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h podName:60335b20-da5c-44f7-8ef1-e1c9858fbff9 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:17.685277032 +0000 UTC m=+65.298320397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zv4h" (UniqueName: "kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h") pod "network-check-target-p6b5k" (UID: "60335b20-da5c-44f7-8ef1-e1c9858fbff9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:46.789820 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:46.789783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:46.790264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:46.789788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:46.790264 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:46.789900    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:46.790264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:46.789946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:46.790264 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:46.790060    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:46.790264 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:46.790131    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:48.789696 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:48.789661    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:48.789696 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:48.789687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:48.790170 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:48.789660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:48.790170 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:48.789781    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6btbd" podUID="59d8dc7f-ebf5-41cb-8951-339301976b04"
Jul 01 19:22:48.790170 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:48.789859    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p6b5k" podUID="60335b20-da5c-44f7-8ef1-e1c9858fbff9"
Jul 01 19:22:48.790170 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:48.789969    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kwwj" podUID="3a87f684-4294-4618-ba06-c37040b80606"
Jul 01 19:22:49.244136 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.244104    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-50.ec2.internal" event="NodeReady"
Jul 01 19:22:49.244304 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.244259    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 19:22:49.352351 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.349808    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86f5bf6777-7wkf7"]
Jul 01 19:22:49.355773 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.355743    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.360037 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.360006    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 19:22:49.360037 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.360029    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 19:22:49.361953 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.361932    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ftr2p\""
Jul 01 19:22:49.368837 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.368805    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 19:22:49.374294 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.374267    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5mdvp"]
Jul 01 19:22:49.377083 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.377062    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 19:22:49.378264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.378243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.383036 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.383017    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 19:22:49.383372 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.383349    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 19:22:49.384110 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.384088    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 19:22:49.388032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.388013    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9btb6\""
Jul 01 19:22:49.403074 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.402086    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vnkct"]
Jul 01 19:22:49.406373 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.406345    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f5bf6777-7wkf7"]
Jul 01 19:22:49.406533 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.406483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.411619 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.411588    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 19:22:49.411780 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.411764    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdgrk\""
Jul 01 19:22:49.411780 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.411595    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 19:22:49.417816 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.417795    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5mdvp"]
Jul 01 19:22:49.435234 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.435196    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vnkct"]
Jul 01 19:22:49.513435 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77s4\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-kube-api-access-v77s4\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.513435 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-trusted-ca\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-bound-sa-token\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37f7468b-37a0-4ab3-b03b-6ff4792847c6-cert\") pod \"ingress-canary-5mdvp\" (UID: \"37f7468b-37a0-4ab3-b03b-6ff4792847c6\") " pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-certificates\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/628e1705-7c95-42b2-a3fd-c524a324c87a-config-volume\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmc7\" (UniqueName: \"kubernetes.io/projected/628e1705-7c95-42b2-a3fd-c524a324c87a-kube-api-access-bzmc7\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm9vn\" (UniqueName: \"kubernetes.io/projected/37f7468b-37a0-4ab3-b03b-6ff4792847c6-kube-api-access-wm9vn\") pod \"ingress-canary-5mdvp\" (UID: \"37f7468b-37a0-4ab3-b03b-6ff4792847c6\") " pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.513694 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-ca-trust-extracted\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.514110 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-image-registry-private-configuration\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.514110 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-installation-pull-secrets\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.514110 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628e1705-7c95-42b2-a3fd-c524a324c87a-tmp-dir\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.514110 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.513784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.614630 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.614599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.614796 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.614648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77s4\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-kube-api-access-v77s4\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.614796 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:49.614771    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:49.614796 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:49.614793    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:22:49.614896 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:49.614880    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:50.114850655 +0000 UTC m=+37.727894098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:22:49.614965 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.614771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-trusted-ca\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.615004 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.614987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-bound-sa-token\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.615044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.615090 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37f7468b-37a0-4ab3-b03b-6ff4792847c6-cert\") pod \"ingress-canary-5mdvp\" (UID: \"37f7468b-37a0-4ab3-b03b-6ff4792847c6\") " pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.615142 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-certificates\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.615192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/628e1705-7c95-42b2-a3fd-c524a324c87a-config-volume\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.615192 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:49.615154    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:49.615288 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmc7\" (UniqueName: \"kubernetes.io/projected/628e1705-7c95-42b2-a3fd-c524a324c87a-kube-api-access-bzmc7\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.615288 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:49.615235    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:22:50.115211617 +0000 UTC m=+37.728254987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:49.615288 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9vn\" (UniqueName: \"kubernetes.io/projected/37f7468b-37a0-4ab3-b03b-6ff4792847c6-kube-api-access-wm9vn\") pod \"ingress-canary-5mdvp\" (UID: \"37f7468b-37a0-4ab3-b03b-6ff4792847c6\") " pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.615449 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-ca-trust-extracted\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.615449 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-image-registry-private-configuration\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.615449 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-installation-pull-secrets\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.615449 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628e1705-7c95-42b2-a3fd-c524a324c87a-tmp-dir\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.616044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628e1705-7c95-42b2-a3fd-c524a324c87a-tmp-dir\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.616044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615804    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-ca-trust-extracted\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.616044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615824    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-certificates\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.616044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.615997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/628e1705-7c95-42b2-a3fd-c524a324c87a-config-volume\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.616299 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.616223    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-trusted-ca\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.619528 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.619505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-installation-pull-secrets\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.619619 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.619511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-image-registry-private-configuration\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.619619 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.619539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37f7468b-37a0-4ab3-b03b-6ff4792847c6-cert\") pod \"ingress-canary-5mdvp\" (UID: \"37f7468b-37a0-4ab3-b03b-6ff4792847c6\") " pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.636040 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.636020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-bound-sa-token\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.636385 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.636361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmc7\" (UniqueName: \"kubernetes.io/projected/628e1705-7c95-42b2-a3fd-c524a324c87a-kube-api-access-bzmc7\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:49.641871 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.641839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77s4\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-kube-api-access-v77s4\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:49.642152 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.642133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm9vn\" (UniqueName: \"kubernetes.io/projected/37f7468b-37a0-4ab3-b03b-6ff4792847c6-kube-api-access-wm9vn\") pod \"ingress-canary-5mdvp\" (UID: \"37f7468b-37a0-4ab3-b03b-6ff4792847c6\") " pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.689050 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.689020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5mdvp"
Jul 01 19:22:49.855653 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.855612    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5mdvp"]
Jul 01 19:22:49.860301 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:22:49.860268    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f7468b_37a0_4ab3_b03b_6ff4792847c6.slice/crio-b377a1dbe4f2e029d1319ca4f7d2e4ab89591bdc72d718b68e8af71a0a8e9f39 WatchSource:0}: Error finding container b377a1dbe4f2e029d1319ca4f7d2e4ab89591bdc72d718b68e8af71a0a8e9f39: Status 404 returned error can't find the container with id b377a1dbe4f2e029d1319ca4f7d2e4ab89591bdc72d718b68e8af71a0a8e9f39
Jul 01 19:22:49.947029 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:49.946989    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pdkcn_4b2bb51a-496e-4929-b481-922ce49e4767/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:50.063950 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.063839    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pdkcn_4b2bb51a-496e-4929-b481-922ce49e4767/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:50.091005 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.090969    2576 generic.go:356] "Generic (PLEG): container finished" podID="256697b6-5404-4d73-94a9-d2363e8c37f1" containerID="756b7b63f9ad51e500d4122784b2e10c9075f9b603652c2dbfaf661c555ed381" exitCode=0
Jul 01 19:22:50.091168 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.091043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerDied","Data":"756b7b63f9ad51e500d4122784b2e10c9075f9b603652c2dbfaf661c555ed381"}
Jul 01 19:22:50.092209 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.092188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5mdvp" event={"ID":"37f7468b-37a0-4ab3-b03b-6ff4792847c6","Type":"ContainerStarted","Data":"b377a1dbe4f2e029d1319ca4f7d2e4ab89591bdc72d718b68e8af71a0a8e9f39"}
Jul 01 19:22:50.120475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.120432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:50.120654 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.120501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:50.120654 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:50.120636    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:50.120654 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:50.120647    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:50.120808 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:50.120668    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:22:50.120808 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:50.120708    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:22:51.120687905 +0000 UTC m=+38.733731274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:50.120808 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:50.120728    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:51.120718853 +0000 UTC m=+38.733762222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:22:50.790132 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.790095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:22:50.790132 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.790126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:22:50.790339 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.790095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:22:50.793420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.793397    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:22:50.793577 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.793397    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:22:50.794900 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.794874    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:22:50.794900 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.794882    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:22:50.795123 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.794927    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4g2f5\""
Jul 01 19:22:50.797557 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:50.797536    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vsxvz\""
Jul 01 19:22:51.099252 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:51.099135    2576 generic.go:356] "Generic (PLEG): container finished" podID="256697b6-5404-4d73-94a9-d2363e8c37f1" containerID="0960b1b9b39e4af7a0e384914c97a0ad52792c87a610bf4e37a7063479a33159" exitCode=0
Jul 01 19:22:51.099252 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:51.099176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerDied","Data":"0960b1b9b39e4af7a0e384914c97a0ad52792c87a610bf4e37a7063479a33159"}
Jul 01 19:22:51.128737 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:51.128496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:51.128908 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:51.128687    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:51.128908 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:51.128854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:51.128908 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:51.128882    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:22:53.128847983 +0000 UTC m=+40.741891379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:51.129094 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:51.128986    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:51.129094 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:51.129002    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:22:51.129094 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:51.129052    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:53.129036942 +0000 UTC m=+40.742080327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:22:52.105875 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:52.105839    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xg25" event={"ID":"256697b6-5404-4d73-94a9-d2363e8c37f1","Type":"ContainerStarted","Data":"76b856c4ec9c3479c7d3e558661193a34d156867b0f9f77f33e6b49ac374b7c5"}
Jul 01 19:22:52.186188 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:52.186122    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8xg25" podStartSLOduration=6.603900496 podStartE2EDuration="39.18610064s" totalImagesPullingTime="32.582200144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:52.179001358 +0000 UTC m=+39.792044747" watchObservedRunningTime="2026-07-01 19:22:52.18610064 +0000 UTC m=+39.799144026"
Jul 01 19:22:53.109365 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:53.109328    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5mdvp" event={"ID":"37f7468b-37a0-4ab3-b03b-6ff4792847c6","Type":"ContainerStarted","Data":"0b1a40fcadf67cd549be3653ee474288221d8bbbd29f4ad038cb7f2e438d7818"}
Jul 01 19:22:53.148234 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:53.148195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:53.148409 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:53.148248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:53.148409 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:53.148329    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:53.148409 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:53.148334    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:53.148409 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:53.148353    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:22:53.148409 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:53.148377    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:22:57.148364857 +0000 UTC m=+44.761408222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:53.148409 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:53.148396    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:57.148381977 +0000 UTC m=+44.761425342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:22:57.177807 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:57.177761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:22:57.178202 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:22:57.177868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:22:57.178202 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:57.177941    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:57.178202 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:57.178003    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:23:05.177989032 +0000 UTC m=+52.791032400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:57.178202 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:57.178018    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:57.178202 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:57.178037    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:22:57.178202 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:22:57.178087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:05.178072645 +0000 UTC m=+52.791116011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:23:05.238063 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:05.238015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:23:05.238547 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:05.238126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:23:05.238547 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:05.238171    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:23:05.238547 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:05.238245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:23:21.238227766 +0000 UTC m=+68.851271130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:23:05.238547 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:05.238252    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:23:05.238547 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:05.238267    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:23:05.238547 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:05.238322    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:21.238306046 +0000 UTC m=+68.851349411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:23:14.083095 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:14.083068    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ppkh"
Jul 01 19:23:14.112587 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:14.112534    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5mdvp" podStartSLOduration=22.646529338 podStartE2EDuration="25.112517184s" totalImagesPullingTime="2.465987846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:53.143955327 +0000 UTC m=+40.756998713" watchObservedRunningTime="2026-07-01 19:23:14.112517184 +0000 UTC m=+61.725560573"
Jul 01 19:23:17.624184 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.624145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:23:17.624603 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.624199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:23:17.626883 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.626862    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:23:17.627030 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.626862    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:23:17.634999 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:17.634980    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 19:23:17.635099 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:17.635037    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs podName:59d8dc7f-ebf5-41cb-8951-339301976b04 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:21.635021684 +0000 UTC m=+129.248065050 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs") pod "network-metrics-daemon-6btbd" (UID: "59d8dc7f-ebf5-41cb-8951-339301976b04") : secret "metrics-daemon-secret" not found
Jul 01 19:23:17.638074 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.638054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a87f684-4294-4618-ba06-c37040b80606-original-pull-secret\") pod \"global-pull-secret-syncer-2kwwj\" (UID: \"3a87f684-4294-4618-ba06-c37040b80606\") " pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:23:17.724987 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.724948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:23:17.727971 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.727949    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:23:17.737834 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.737806    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:23:17.748370 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.748340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv4h\" (UniqueName: \"kubernetes.io/projected/60335b20-da5c-44f7-8ef1-e1c9858fbff9-kube-api-access-4zv4h\") pod \"network-check-target-p6b5k\" (UID: \"60335b20-da5c-44f7-8ef1-e1c9858fbff9\") " pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:23:17.802361 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.802326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kwwj"
Jul 01 19:23:17.813208 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.813179    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4g2f5\""
Jul 01 19:23:17.820542 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.820520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:23:17.954067 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.954036    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kwwj"]
Jul 01 19:23:17.957715 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:23:17.957691    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a87f684_4294_4618_ba06_c37040b80606.slice/crio-4c7ee73e446f0a37065d5d88306e918d77900a5606f5170d1a45eed6d2e29364 WatchSource:0}: Error finding container 4c7ee73e446f0a37065d5d88306e918d77900a5606f5170d1a45eed6d2e29364: Status 404 returned error can't find the container with id 4c7ee73e446f0a37065d5d88306e918d77900a5606f5170d1a45eed6d2e29364
Jul 01 19:23:17.971927 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:17.971889    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p6b5k"]
Jul 01 19:23:17.975330 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:23:17.975303    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60335b20_da5c_44f7_8ef1_e1c9858fbff9.slice/crio-3c27b0c8753a82803a1fece7d210a69d86760e931afefe896fccf99a8134117d WatchSource:0}: Error finding container 3c27b0c8753a82803a1fece7d210a69d86760e931afefe896fccf99a8134117d: Status 404 returned error can't find the container with id 3c27b0c8753a82803a1fece7d210a69d86760e931afefe896fccf99a8134117d
Jul 01 19:23:18.166716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:18.166626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p6b5k" event={"ID":"60335b20-da5c-44f7-8ef1-e1c9858fbff9","Type":"ContainerStarted","Data":"3c27b0c8753a82803a1fece7d210a69d86760e931afefe896fccf99a8134117d"}
Jul 01 19:23:18.167798 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:18.167769    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2kwwj" event={"ID":"3a87f684-4294-4618-ba06-c37040b80606","Type":"ContainerStarted","Data":"4c7ee73e446f0a37065d5d88306e918d77900a5606f5170d1a45eed6d2e29364"}
Jul 01 19:23:21.256543 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:21.254775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:23:21.256543 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:21.254868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:23:21.256543 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:21.255058    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:23:21.256543 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:21.255119    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:23:53.255101532 +0000 UTC m=+100.868144914 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:23:21.256543 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:21.255651    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:23:21.256543 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:21.255670    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:23:21.256543 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:21.255722    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:53.2557055 +0000 UTC m=+100.868748888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:23:23.183129 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:23.183096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p6b5k" event={"ID":"60335b20-da5c-44f7-8ef1-e1c9858fbff9","Type":"ContainerStarted","Data":"24fb5fe7965edac3f4670287e7df0663ccd9e08bdc67f935c60e6709ef05c57d"}
Jul 01 19:23:23.183553 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:23.183222    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:23:23.184596 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:23.184573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2kwwj" event={"ID":"3a87f684-4294-4618-ba06-c37040b80606","Type":"ContainerStarted","Data":"56bbe919b17317064921f75cc1f320861cc9f422ac63d1b271e3900dfca69ab1"}
Jul 01 19:23:23.203182 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:23.203139    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p6b5k" podStartSLOduration=65.474303596 podStartE2EDuration="1m10.20312437s" totalImagesPullingTime="4.728820774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:23.202479849 +0000 UTC m=+70.815523237" watchObservedRunningTime="2026-07-01 19:23:23.20312437 +0000 UTC m=+70.816167735"
Jul 01 19:23:23.219185 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:23.219138    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2kwwj" podStartSLOduration=65.461434938 podStartE2EDuration="1m10.219122876s" totalImagesPullingTime="4.757687938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:23.218858087 +0000 UTC m=+70.831901475" watchObservedRunningTime="2026-07-01 19:23:23.219122876 +0000 UTC m=+70.832166260"
Jul 01 19:23:38.247665 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.247557    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"]
Jul 01 19:23:38.251771 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.251753    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.258190 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.258126    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 01 19:23:38.258718 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.258703    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gcjsz\""
Jul 01 19:23:38.259307 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.259292    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 01 19:23:38.260893 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.260873    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 01 19:23:38.265870 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.265852    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:23:38.288047 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.288019    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"]
Jul 01 19:23:38.377788 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.377745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2a85bfa-282d-4472-94ca-4811bb44335b-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.377788 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.377784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2a85bfa-282d-4472-94ca-4811bb44335b-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.378038 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.377812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4sr\" (UniqueName: \"kubernetes.io/projected/e2a85bfa-282d-4472-94ca-4811bb44335b-kube-api-access-zf4sr\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.479081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.479049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2a85bfa-282d-4472-94ca-4811bb44335b-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.479081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.479083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2a85bfa-282d-4472-94ca-4811bb44335b-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.479228 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.479112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4sr\" (UniqueName: \"kubernetes.io/projected/e2a85bfa-282d-4472-94ca-4811bb44335b-kube-api-access-zf4sr\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.479738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.479720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2a85bfa-282d-4472-94ca-4811bb44335b-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.481282 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.481265    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2a85bfa-282d-4472-94ca-4811bb44335b-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.487326 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.487296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4sr\" (UniqueName: \"kubernetes.io/projected/e2a85bfa-282d-4472-94ca-4811bb44335b-kube-api-access-zf4sr\") pod \"kube-storage-version-migrator-operator-79964ff6f9-jjrkw\" (UID: \"e2a85bfa-282d-4472-94ca-4811bb44335b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.563535 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.563489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"
Jul 01 19:23:38.687986 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:38.687956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw"]
Jul 01 19:23:38.691143 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:23:38.691111    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a85bfa_282d_4472_94ca_4811bb44335b.slice/crio-a6a627260bb34b88a59c74ee06ab4381c2e9aca42e8ad411e34f97822aefd458 WatchSource:0}: Error finding container a6a627260bb34b88a59c74ee06ab4381c2e9aca42e8ad411e34f97822aefd458: Status 404 returned error can't find the container with id a6a627260bb34b88a59c74ee06ab4381c2e9aca42e8ad411e34f97822aefd458
Jul 01 19:23:39.224493 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:39.224460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw" event={"ID":"e2a85bfa-282d-4472-94ca-4811bb44335b","Type":"ContainerStarted","Data":"a6a627260bb34b88a59c74ee06ab4381c2e9aca42e8ad411e34f97822aefd458"}
Jul 01 19:23:41.553745 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.553712    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dgnqs"]
Jul 01 19:23:41.556692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.556672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.559265 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.559240    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 19:23:41.559386 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.559303    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 19:23:41.559519 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.559487    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 19:23:41.559568 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.559551    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xh89l\""
Jul 01 19:23:41.560510 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.560492    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 19:23:41.568584 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.568562    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dgnqs"]
Jul 01 19:23:41.603726 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.603689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lbm\" (UniqueName: \"kubernetes.io/projected/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-api-access-b9lbm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.603726 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.603734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.603998 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.603755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.603998 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.603836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-data-volume\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.603998 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.603976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-crio-socket\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.705303 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.705263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lbm\" (UniqueName: \"kubernetes.io/projected/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-api-access-b9lbm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.705473 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.705313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.705473 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:41.705386    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:23:41.705473 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:41.705470    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:42.205450477 +0000 UTC m=+89.818493848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : configmap "kube-rbac-proxy" not found
Jul 01 19:23:41.705623 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.705491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.705623 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.705541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-data-volume\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.705623 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.705607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-crio-socket\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.705748 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:41.705617    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:41.705748 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:41.705678    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:42.205658782 +0000 UTC m=+89.818702165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:41.705748 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.705698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-crio-socket\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.705955 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.705938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-data-volume\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:41.716398 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:41.716359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lbm\" (UniqueName: \"kubernetes.io/projected/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-api-access-b9lbm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:42.209964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:42.209908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:42.209964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:42.209961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:42.210186 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:42.210043    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:23:42.210186 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:42.210076    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:42.210186 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:42.210110    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:43.210095585 +0000 UTC m=+90.823138949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : configmap "kube-rbac-proxy" not found
Jul 01 19:23:42.210186 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:42.210124    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:43.21011789 +0000 UTC m=+90.823161254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:42.233297 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:42.233263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw" event={"ID":"e2a85bfa-282d-4472-94ca-4811bb44335b","Type":"ContainerStarted","Data":"cd3a06d25603e9f03605cf50b9afe2360d3237b48841f394f0fe68a796b662b0"}
Jul 01 19:23:43.217738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:43.217686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:43.217738 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:43.217735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:43.218327 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:43.217814    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:23:43.218327 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:43.217880    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:45.217863744 +0000 UTC m=+92.830907108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : configmap "kube-rbac-proxy" not found
Jul 01 19:23:43.218327 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:43.217879    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:43.218327 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:43.217907    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:45.217900924 +0000 UTC m=+92.830944288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:43.324720 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:43.324694    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7ktbn_3023be20-3d3a-4833-914f-1f98ea4ba5ee/dns-node-resolver/0.log"
Jul 01 19:23:44.125335 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:44.125306    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cg7l_b755486f-bcce-43d8-b6d7-c06e780555fe/node-ca/0.log"
Jul 01 19:23:45.126132 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:45.126100    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5mdvp_37f7468b-37a0-4ab3-b03b-6ff4792847c6/serve-healthcheck-canary/0.log"
Jul 01 19:23:45.233122 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:45.233086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:45.233122 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:45.233120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:45.233376 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:45.233228    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:23:45.233376 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:45.233300    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:45.233376 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:45.233303    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.233282581 +0000 UTC m=+96.846325964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : configmap "kube-rbac-proxy" not found
Jul 01 19:23:45.233376 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:45.233357    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.233346006 +0000 UTC m=+96.846389371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:47.466254 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.466197    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw" podStartSLOduration=6.109514704 podStartE2EDuration="9.466179604s" totalImagesPullingTime="3.3566649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:42.257458768 +0000 UTC m=+89.870502156" watchObservedRunningTime="2026-07-01 19:23:47.466179604 +0000 UTC m=+95.079222991"
Jul 01 19:23:47.466676 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.466601    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"]
Jul 01 19:23:47.469638 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.469623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:47.477543 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.477520    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 19:23:47.481533 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.481490    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 19:23:47.481645 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.481614    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dt6pp\""
Jul 01 19:23:47.497605 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.497569    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"]
Jul 01 19:23:47.550086 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.550047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:47.550248 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.550094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a110deee-26de-412e-a746-2bbd075dd8c2-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:47.650780 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.650731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:47.651005 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.650788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a110deee-26de-412e-a746-2bbd075dd8c2-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:47.651005 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:47.650886    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:23:47.651005 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:47.650978    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert podName:a110deee-26de-412e-a746-2bbd075dd8c2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:48.150961723 +0000 UTC m=+95.764005088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-kqc77" (UID: "a110deee-26de-412e-a746-2bbd075dd8c2") : secret "networking-console-plugin-cert" not found
Jul 01 19:23:47.651476 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:47.651453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a110deee-26de-412e-a746-2bbd075dd8c2-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:48.154387 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:48.154330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:48.154576 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:48.154485    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:23:48.154576 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:48.154548    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert podName:a110deee-26de-412e-a746-2bbd075dd8c2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:49.154534037 +0000 UTC m=+96.767577402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-kqc77" (UID: "a110deee-26de-412e-a746-2bbd075dd8c2") : secret "networking-console-plugin-cert" not found
Jul 01 19:23:49.162869 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:49.162822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:49.163307 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:49.163003    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:23:49.163307 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:49.163076    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert podName:a110deee-26de-412e-a746-2bbd075dd8c2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:51.163061876 +0000 UTC m=+98.776105240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-kqc77" (UID: "a110deee-26de-412e-a746-2bbd075dd8c2") : secret "networking-console-plugin-cert" not found
Jul 01 19:23:49.263778 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:49.263746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:49.263778 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:49.263779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:49.263982 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:49.263888    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:49.263982 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:49.263941    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:23:49.264049 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:49.263953    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:57.263938412 +0000 UTC m=+104.876981793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:49.264049 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:49.264017    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:57.264003543 +0000 UTC m=+104.877046912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : configmap "kube-rbac-proxy" not found
Jul 01 19:23:51.179976 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:51.179944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:51.180428 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:51.180089    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:23:51.180428 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:51.180157    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert podName:a110deee-26de-412e-a746-2bbd075dd8c2 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.180142907 +0000 UTC m=+102.793186272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-kqc77" (UID: "a110deee-26de-412e-a746-2bbd075dd8c2") : secret "networking-console-plugin-cert" not found
Jul 01 19:23:53.295740 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:53.295684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") pod \"image-registry-86f5bf6777-7wkf7\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") " pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:23:53.296216 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:53.295790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:23:53.296216 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:53.295837    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:23:53.296216 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:53.295859    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f5bf6777-7wkf7: secret "image-registry-tls" not found
Jul 01 19:23:53.296216 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:53.295942    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls podName:c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:57.29590739 +0000 UTC m=+164.908950755 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls") pod "image-registry-86f5bf6777-7wkf7" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2") : secret "image-registry-tls" not found
Jul 01 19:23:53.296216 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:53.295959    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:23:53.296216 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:53.296020    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls podName:628e1705-7c95-42b2-a3fd-c524a324c87a nodeName:}" failed. No retries permitted until 2026-07-01 19:24:57.296007304 +0000 UTC m=+164.909050669 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls") pod "dns-default-vnkct" (UID: "628e1705-7c95-42b2-a3fd-c524a324c87a") : secret "dns-default-metrics-tls" not found
Jul 01 19:23:54.190019 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:54.189991    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p6b5k"
Jul 01 19:23:55.212003 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:55.211961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:23:55.212389 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:55.212115    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:23:55.212389 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:55.212194    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert podName:a110deee-26de-412e-a746-2bbd075dd8c2 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.21217767 +0000 UTC m=+110.825221035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-kqc77" (UID: "a110deee-26de-412e-a746-2bbd075dd8c2") : secret "networking-console-plugin-cert" not found
Jul 01 19:23:57.331014 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:57.330979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:57.331014 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:57.331014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:23:57.331438 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:57.331113    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:57.331438 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:23:57.331166    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls podName:a7fdf84c-9fdd-43e8-8200-cb82e5b73262 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:13.331151724 +0000 UTC m=+120.944195089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dgnqs" (UID: "a7fdf84c-9fdd-43e8-8200-cb82e5b73262") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:57.331530 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:23:57.331467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:24:03.278950 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:03.278880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:24:03.281242 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:03.281221    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a110deee-26de-412e-a746-2bbd075dd8c2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-kqc77\" (UID: \"a110deee-26de-412e-a746-2bbd075dd8c2\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:24:03.378493 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:03.378444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"
Jul 01 19:24:03.503585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:03.503551    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77"]
Jul 01 19:24:03.506299 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:03.506268    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda110deee_26de_412e_a746_2bbd075dd8c2.slice/crio-213b32ed5360f6f35fed421d32073e7aabc9bf26202614e73ca9d2d8c78fba30 WatchSource:0}: Error finding container 213b32ed5360f6f35fed421d32073e7aabc9bf26202614e73ca9d2d8c78fba30: Status 404 returned error can't find the container with id 213b32ed5360f6f35fed421d32073e7aabc9bf26202614e73ca9d2d8c78fba30
Jul 01 19:24:04.287720 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:04.287687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77" event={"ID":"a110deee-26de-412e-a746-2bbd075dd8c2","Type":"ContainerStarted","Data":"213b32ed5360f6f35fed421d32073e7aabc9bf26202614e73ca9d2d8c78fba30"}
Jul 01 19:24:05.292045 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:05.292005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77" event={"ID":"a110deee-26de-412e-a746-2bbd075dd8c2","Type":"ContainerStarted","Data":"3ac4723de62da79a4dd23c4cb338979b07d8e37540a2f8d8446094aab2d79ca6"}
Jul 01 19:24:05.311567 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:05.311455    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-kqc77" podStartSLOduration=17.098985958 podStartE2EDuration="18.311435764s" totalImagesPullingTime="1.212449806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:05.310874724 +0000 UTC m=+112.923918121" watchObservedRunningTime="2026-07-01 19:24:05.311435764 +0000 UTC m=+112.924479152"
Jul 01 19:24:07.436514 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.436479    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"]
Jul 01 19:24:07.439709 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.439688    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"]
Jul 01 19:24:07.439823 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.439812    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.442847 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.442824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.442988 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.442826    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 19:24:07.443091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.443070    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 19:24:07.443141 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.442831    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 19:24:07.443180 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.442828    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 19:24:07.445349 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.445330    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 19:24:07.445668 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.445650    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 19:24:07.446249 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.446230    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 19:24:07.446418 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.446406    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 19:24:07.449937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.449900    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"]
Jul 01 19:24:07.457428 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.457407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"]
Jul 01 19:24:07.608294 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49c1d085-f7fd-42fd-ac08-9264772c9580-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.608475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c1d085-f7fd-42fd-ac08-9264772c9580-tmp\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.608475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-hub\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.608475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flp64\" (UniqueName: \"kubernetes.io/projected/46e495f7-2b73-4ecf-b421-3dec6d0ec413-kube-api-access-flp64\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.608722 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e495f7-2b73-4ecf-b421-3dec6d0ec413-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.608722 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.608722 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttsmm\" (UniqueName: \"kubernetes.io/projected/49c1d085-f7fd-42fd-ac08-9264772c9580-kube-api-access-ttsmm\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.608722 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-ca\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.608722 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.608698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.709149 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c1d085-f7fd-42fd-ac08-9264772c9580-tmp\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.709149 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-hub\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.709149 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flp64\" (UniqueName: \"kubernetes.io/projected/46e495f7-2b73-4ecf-b421-3dec6d0ec413-kube-api-access-flp64\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.709420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709170    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e495f7-2b73-4ecf-b421-3dec6d0ec413-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.709420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.709420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttsmm\" (UniqueName: \"kubernetes.io/projected/49c1d085-f7fd-42fd-ac08-9264772c9580-kube-api-access-ttsmm\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.709420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-ca\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.709420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.709420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49c1d085-f7fd-42fd-ac08-9264772c9580-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.709686 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.709530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c1d085-f7fd-42fd-ac08-9264772c9580-tmp\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.710104 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.710083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e495f7-2b73-4ecf-b421-3dec6d0ec413-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.711859 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.711835    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.711995 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.711978    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49c1d085-f7fd-42fd-ac08-9264772c9580-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.712187 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.712170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-hub\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.712245 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.712225    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.712363 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.712347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/46e495f7-2b73-4ecf-b421-3dec6d0ec413-ca\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.720900 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.720872    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flp64\" (UniqueName: \"kubernetes.io/projected/46e495f7-2b73-4ecf-b421-3dec6d0ec413-kube-api-access-flp64\") pod \"cluster-proxy-proxy-agent-7cc4b9546f-qr6mb\" (UID: \"46e495f7-2b73-4ecf-b421-3dec6d0ec413\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.721406 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.721388    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttsmm\" (UniqueName: \"kubernetes.io/projected/49c1d085-f7fd-42fd-ac08-9264772c9580-kube-api-access-ttsmm\") pod \"klusterlet-addon-workmgr-dc49b949c-7lv5s\" (UID: \"49c1d085-f7fd-42fd-ac08-9264772c9580\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.750565 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.750521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:07.762772 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.762741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"
Jul 01 19:24:07.892493 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.892443    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"]
Jul 01 19:24:07.894862 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:07.894833    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c1d085_f7fd_42fd_ac08_9264772c9580.slice/crio-16b1133550330e459bdd1548879549109082ebedc4417bd958ad9a329fe3aadc WatchSource:0}: Error finding container 16b1133550330e459bdd1548879549109082ebedc4417bd958ad9a329fe3aadc: Status 404 returned error can't find the container with id 16b1133550330e459bdd1548879549109082ebedc4417bd958ad9a329fe3aadc
Jul 01 19:24:07.910411 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:07.910392    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb"]
Jul 01 19:24:07.912714 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:07.912681    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e495f7_2b73_4ecf_b421_3dec6d0ec413.slice/crio-7d22499bd49d0accb35a58fd9f47d71b2407bb6400f82a4c85b29a6dcf438710 WatchSource:0}: Error finding container 7d22499bd49d0accb35a58fd9f47d71b2407bb6400f82a4c85b29a6dcf438710: Status 404 returned error can't find the container with id 7d22499bd49d0accb35a58fd9f47d71b2407bb6400f82a4c85b29a6dcf438710
Jul 01 19:24:08.299459 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:08.299425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb" event={"ID":"46e495f7-2b73-4ecf-b421-3dec6d0ec413","Type":"ContainerStarted","Data":"7d22499bd49d0accb35a58fd9f47d71b2407bb6400f82a4c85b29a6dcf438710"}
Jul 01 19:24:08.300433 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:08.300410    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s" event={"ID":"49c1d085-f7fd-42fd-ac08-9264772c9580","Type":"ContainerStarted","Data":"16b1133550330e459bdd1548879549109082ebedc4417bd958ad9a329fe3aadc"}
Jul 01 19:24:13.356164 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:13.356114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:24:13.358623 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:13.358595    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7fdf84c-9fdd-43e8-8200-cb82e5b73262-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dgnqs\" (UID: \"a7fdf84c-9fdd-43e8-8200-cb82e5b73262\") " pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:24:13.369370 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:13.369337    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xh89l\""
Jul 01 19:24:13.377019 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:13.376991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dgnqs"
Jul 01 19:24:14.468051 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:14.467936    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dgnqs"]
Jul 01 19:24:14.475495 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:14.475012    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fdf84c_9fdd_43e8_8200_cb82e5b73262.slice/crio-34c6afed149feef7ac4924ee067638381d50d7ad23cb8472f32005b4598bd276 WatchSource:0}: Error finding container 34c6afed149feef7ac4924ee067638381d50d7ad23cb8472f32005b4598bd276: Status 404 returned error can't find the container with id 34c6afed149feef7ac4924ee067638381d50d7ad23cb8472f32005b4598bd276
Jul 01 19:24:15.325947 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.325889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dgnqs" event={"ID":"a7fdf84c-9fdd-43e8-8200-cb82e5b73262","Type":"ContainerStarted","Data":"34c6afed149feef7ac4924ee067638381d50d7ad23cb8472f32005b4598bd276"}
Jul 01 19:24:15.328648 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.328603    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb" event={"ID":"46e495f7-2b73-4ecf-b421-3dec6d0ec413","Type":"ContainerStarted","Data":"4794139a83bf0e4726aec04792e7da265fd22ef1a228855746ecc005362ffd76"}
Jul 01 19:24:15.328804 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.328663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb" event={"ID":"46e495f7-2b73-4ecf-b421-3dec6d0ec413","Type":"ContainerStarted","Data":"0953a3daeaf1474a336fdb7d3eb6e142cb86d169e21900a1a1b953221a104283"}
Jul 01 19:24:15.328804 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.328681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb" event={"ID":"46e495f7-2b73-4ecf-b421-3dec6d0ec413","Type":"ContainerStarted","Data":"66e1ede841ed213dd7839dab9f2aa5161aaca1e1f7a2a3223b2a91033ace3f77"}
Jul 01 19:24:15.330489 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.330460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s" event={"ID":"49c1d085-f7fd-42fd-ac08-9264772c9580","Type":"ContainerStarted","Data":"2ca8b3ea2b8ee30b9993c10345fb31b054168844c5eaa9b7afd100b145dd93c7"}
Jul 01 19:24:15.330906 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.330882    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:15.332858 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.332837    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s"
Jul 01 19:24:15.349787 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.349741    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc4b9546f-qr6mb" podStartSLOduration=1.91721143 podStartE2EDuration="8.349719214s" totalImagesPullingTime="6.432507784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:15.348621695 +0000 UTC m=+122.961665082" watchObservedRunningTime="2026-07-01 19:24:15.349719214 +0000 UTC m=+122.962762605"
Jul 01 19:24:15.366491 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:15.366429    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc49b949c-7lv5s" podStartSLOduration=1.925330577 podStartE2EDuration="8.36640897s" totalImagesPullingTime="6.441078393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:15.365898523 +0000 UTC m=+122.978941914" watchObservedRunningTime="2026-07-01 19:24:15.36640897 +0000 UTC m=+122.979452358"
Jul 01 19:24:19.344948 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:19.344900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dgnqs" event={"ID":"a7fdf84c-9fdd-43e8-8200-cb82e5b73262","Type":"ContainerStarted","Data":"04a6cb2a87e50ff7735e495ed9a2f224fb5fdd3703ac4558c6129f4146d3f24b"}
Jul 01 19:24:19.345307 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:19.344967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dgnqs" event={"ID":"a7fdf84c-9fdd-43e8-8200-cb82e5b73262","Type":"ContainerStarted","Data":"07afdd4d75cdef23c4a93a105b14f44f845d10f7037482790108bad663f9c121"}
Jul 01 19:24:21.719268 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:21.719233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:24:21.721463 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:21.721438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59d8dc7f-ebf5-41cb-8951-339301976b04-metrics-certs\") pod \"network-metrics-daemon-6btbd\" (UID: \"59d8dc7f-ebf5-41cb-8951-339301976b04\") " pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:24:22.018235 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:22.018204    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vsxvz\""
Jul 01 19:24:22.026415 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:22.026385    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6btbd"
Jul 01 19:24:22.150068 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:22.150037    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6btbd"]
Jul 01 19:24:22.152792 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:22.152763    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d8dc7f_ebf5_41cb_8951_339301976b04.slice/crio-9046a1fbc8fcee970664ee97857f5e1e79905f901bf2b52892050a99fa72d784 WatchSource:0}: Error finding container 9046a1fbc8fcee970664ee97857f5e1e79905f901bf2b52892050a99fa72d784: Status 404 returned error can't find the container with id 9046a1fbc8fcee970664ee97857f5e1e79905f901bf2b52892050a99fa72d784
Jul 01 19:24:22.357530 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:22.357437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dgnqs" event={"ID":"a7fdf84c-9fdd-43e8-8200-cb82e5b73262","Type":"ContainerStarted","Data":"cbeb54e5d91ab65e18dc08d74351a1d954e2f0b96566c43e7e46fb2d49933ab3"}
Jul 01 19:24:22.358539 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:22.358513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6btbd" event={"ID":"59d8dc7f-ebf5-41cb-8951-339301976b04","Type":"ContainerStarted","Data":"9046a1fbc8fcee970664ee97857f5e1e79905f901bf2b52892050a99fa72d784"}
Jul 01 19:24:22.382513 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:22.379612    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dgnqs" podStartSLOduration=34.158915586 podStartE2EDuration="41.379593109s" totalImagesPullingTime="7.220677523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:22.378335722 +0000 UTC m=+129.991379120" watchObservedRunningTime="2026-07-01 19:24:22.379593109 +0000 UTC m=+129.992636496"
Jul 01 19:24:23.084261 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.084228    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7r6jz"]
Jul 01 19:24:23.088681 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.088652    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.091823 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.091799    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 19:24:23.093090 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.093066    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 19:24:23.093090 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.093087    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bfs88\""
Jul 01 19:24:23.093245 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.093112    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 19:24:23.093245 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.093238    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 19:24:23.093346 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.093328    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 19:24:23.093451 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.093431    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 19:24:23.132547 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-tls\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-root\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-wtmp\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-metrics-client-ca\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjt8t\" (UniqueName: \"kubernetes.io/projected/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-kube-api-access-hjt8t\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-sys\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132900 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-textfile\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.132900 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.132763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233388 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-root\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233526 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-wtmp\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233526 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233526 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-metrics-client-ca\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233526 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjt8t\" (UniqueName: \"kubernetes.io/projected/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-kube-api-access-hjt8t\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233526 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-root\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233528    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-wtmp\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-sys\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-textfile\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233880 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-sys\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.233880 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.233719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-tls\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.234013 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:24:23.233877    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 19:24:23.234013 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:24:23.233956    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-tls podName:24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e nodeName:}" failed. No retries permitted until 2026-07-01 19:24:23.73393822 +0000 UTC m=+131.346981586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-tls") pod "node-exporter-7r6jz" (UID: "24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e") : secret "node-exporter-tls" not found
Jul 01 19:24:23.234115 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.234028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-metrics-client-ca\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.234115 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.234032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-textfile\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.234233 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.234215    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.235696 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.235678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.241759 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.241734    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjt8t\" (UniqueName: \"kubernetes.io/projected/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-kube-api-access-hjt8t\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.364107 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.364070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6btbd" event={"ID":"59d8dc7f-ebf5-41cb-8951-339301976b04","Type":"ContainerStarted","Data":"85e97dcffb96b8d83f2dfee6c009fa32b9f8dfe7403129dcadd1c004eabd592b"}
Jul 01 19:24:23.364256 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.364123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6btbd" event={"ID":"59d8dc7f-ebf5-41cb-8951-339301976b04","Type":"ContainerStarted","Data":"bbaac2d841f02fd7ceaad71d7f129417bf8ef9e530382fa2d1d800b744ea99d7"}
Jul 01 19:24:23.384689 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.384636    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6btbd" podStartSLOduration=129.375056095 podStartE2EDuration="2m10.384619802s" totalImagesPullingTime="1.009563707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:23.383785528 +0000 UTC m=+130.996828930" watchObservedRunningTime="2026-07-01 19:24:23.384619802 +0000 UTC m=+130.997663189"
Jul 01 19:24:23.738088 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.737988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-tls\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:23.740302 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:23.740280    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e-node-exporter-tls\") pod \"node-exporter-7r6jz\" (UID: \"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e\") " pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:24.000625 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:24.000536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7r6jz"
Jul 01 19:24:24.011135 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:24.011101    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b64bcb_2e76_48f8_bdb8_e1df1f6dd62e.slice/crio-d62f085aab39970e5da002e14bae8fe74ec4403dc180e3035002c7773d80350a WatchSource:0}: Error finding container d62f085aab39970e5da002e14bae8fe74ec4403dc180e3035002c7773d80350a: Status 404 returned error can't find the container with id d62f085aab39970e5da002e14bae8fe74ec4403dc180e3035002c7773d80350a
Jul 01 19:24:24.371319 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:24.371279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r6jz" event={"ID":"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e","Type":"ContainerStarted","Data":"d62f085aab39970e5da002e14bae8fe74ec4403dc180e3035002c7773d80350a"}
Jul 01 19:24:25.375782 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:25.375736    2576 generic.go:356] "Generic (PLEG): container finished" podID="24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e" containerID="619e33c1b3bdf6c9c123347bc2ac9d8a90de6be72eae36a77fb3084fadc979f5" exitCode=0
Jul 01 19:24:25.376136 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:25.375817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r6jz" event={"ID":"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e","Type":"ContainerDied","Data":"619e33c1b3bdf6c9c123347bc2ac9d8a90de6be72eae36a77fb3084fadc979f5"}
Jul 01 19:24:26.383334 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:26.383289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r6jz" event={"ID":"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e","Type":"ContainerStarted","Data":"98493324911b163d18873df79d98d29cf8e821bb7f263c577127edad7270aa2a"}
Jul 01 19:24:26.383827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:26.383364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r6jz" event={"ID":"24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e","Type":"ContainerStarted","Data":"7391ff645d1f4328b61a7787ca857d64d33a0440011215dcb2b4077a5589a26e"}
Jul 01 19:24:26.406242 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:26.406192    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7r6jz" podStartSLOduration=2.173889858 podStartE2EDuration="3.406174557s" totalImagesPullingTime="1.16898643s" totalInitContainerRuntime="63.298269ms" isStatefulPod=true podCreationTimestamp="2026-07-01 19:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:26.404979195 +0000 UTC m=+134.018022586" watchObservedRunningTime="2026-07-01 19:24:26.406174557 +0000 UTC m=+134.019217978"
Jul 01 19:24:29.850825 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:29.850789    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f5bf6777-7wkf7"]
Jul 01 19:24:29.851221 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:24:29.851038    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7" podUID="c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"
Jul 01 19:24:30.395338 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.395307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:24:30.401872 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.401846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:24:30.490302 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490263    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-bound-sa-token\") pod \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") "
Jul 01 19:24:30.490475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490317    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-ca-trust-extracted\") pod \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") "
Jul 01 19:24:30.490475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490342    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-certificates\") pod \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") "
Jul 01 19:24:30.490475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490375    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-image-registry-private-configuration\") pod \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") "
Jul 01 19:24:30.490475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490420    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-installation-pull-secrets\") pod \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") "
Jul 01 19:24:30.490475 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490446    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v77s4\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-kube-api-access-v77s4\") pod \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") "
Jul 01 19:24:30.490708 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490484    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-trusted-ca\") pod \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\" (UID: \"c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2\") "
Jul 01 19:24:30.490763 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490706    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:24:30.490813 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.490760    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:30.491064 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.491033    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:30.492747 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.492716    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:30.492747 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.492739    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:30.492984 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.492798    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:30.492984 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.492822    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-kube-api-access-v77s4" (OuterVolumeSpecName: "kube-api-access-v77s4") pod "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" (UID: "c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2"). InnerVolumeSpecName "kube-api-access-v77s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:30.591187 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.591145    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-trusted-ca\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:30.591187 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.591175    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-bound-sa-token\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:30.591187 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.591185    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-ca-trust-extracted\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:30.591187 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.591194    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-certificates\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:30.591501 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.591203    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-image-registry-private-configuration\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:30.591501 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.591213    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-installation-pull-secrets\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:30.591501 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:30.591222    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77s4\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-kube-api-access-v77s4\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:31.398600 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.398568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f5bf6777-7wkf7"
Jul 01 19:24:31.430716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.430681    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f5bf6777-7wkf7"]
Jul 01 19:24:31.436460 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.436434    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86f5bf6777-7wkf7"]
Jul 01 19:24:31.498929 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.498889    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2-registry-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:24:31.626771 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.626739    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-vdvph"]
Jul 01 19:24:31.630909 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.630890    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-vdvph"
Jul 01 19:24:31.633592 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.633563    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 19:24:31.633592 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.633585    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c8n6n\""
Jul 01 19:24:31.633765 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.633735    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 19:24:31.639612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.639592    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-vdvph"]
Jul 01 19:24:31.701464 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.701368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lj7\" (UniqueName: \"kubernetes.io/projected/a59f23c7-910a-47a4-8350-067355cc0aec-kube-api-access-q5lj7\") pod \"downloads-5499b6fb8f-vdvph\" (UID: \"a59f23c7-910a-47a4-8350-067355cc0aec\") " pod="openshift-console/downloads-5499b6fb8f-vdvph"
Jul 01 19:24:31.802219 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.802183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lj7\" (UniqueName: \"kubernetes.io/projected/a59f23c7-910a-47a4-8350-067355cc0aec-kube-api-access-q5lj7\") pod \"downloads-5499b6fb8f-vdvph\" (UID: \"a59f23c7-910a-47a4-8350-067355cc0aec\") " pod="openshift-console/downloads-5499b6fb8f-vdvph"
Jul 01 19:24:31.811437 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.811411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lj7\" (UniqueName: \"kubernetes.io/projected/a59f23c7-910a-47a4-8350-067355cc0aec-kube-api-access-q5lj7\") pod \"downloads-5499b6fb8f-vdvph\" (UID: \"a59f23c7-910a-47a4-8350-067355cc0aec\") " pod="openshift-console/downloads-5499b6fb8f-vdvph"
Jul 01 19:24:31.940827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:31.940789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-vdvph"
Jul 01 19:24:32.067138 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:32.066962    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-vdvph"]
Jul 01 19:24:32.069633 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:32.069604    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59f23c7_910a_47a4_8350_067355cc0aec.slice/crio-d2fefcc1b253da4da9096830b2705ca284dc75bc109bb29f1cf97ca0df57e45a WatchSource:0}: Error finding container d2fefcc1b253da4da9096830b2705ca284dc75bc109bb29f1cf97ca0df57e45a: Status 404 returned error can't find the container with id d2fefcc1b253da4da9096830b2705ca284dc75bc109bb29f1cf97ca0df57e45a
Jul 01 19:24:32.403512 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:32.403480    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-vdvph" event={"ID":"a59f23c7-910a-47a4-8350-067355cc0aec","Type":"ContainerStarted","Data":"d2fefcc1b253da4da9096830b2705ca284dc75bc109bb29f1cf97ca0df57e45a"}
Jul 01 19:24:32.797598 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:32.795551    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2" path="/var/lib/kubelet/pods/c57fd7cb-7140-4a0b-96cf-68ed2ddcdfc2/volumes"
Jul 01 19:24:50.468404 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:50.468350    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-vdvph" event={"ID":"a59f23c7-910a-47a4-8350-067355cc0aec","Type":"ContainerStarted","Data":"e03c2befd0e36b14289c589b45ec38e35c6f60f6d42b9ff9a6eb7e1802ee0d3b"}
Jul 01 19:24:50.468935 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:50.468565    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-vdvph"
Jul 01 19:24:50.485549 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:50.485517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-vdvph"
Jul 01 19:24:50.493644 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:50.493588    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-vdvph" podStartSLOduration=2.176956643 podStartE2EDuration="19.493570975s" totalImagesPullingTime="17.316614332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:50.491726351 +0000 UTC m=+158.104769764" watchObservedRunningTime="2026-07-01 19:24:50.493570975 +0000 UTC m=+158.106614363"
Jul 01 19:24:52.416935 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:24:52.416878    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vnkct" podUID="628e1705-7c95-42b2-a3fd-c524a324c87a"
Jul 01 19:24:52.476812 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:52.476776    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vnkct"
Jul 01 19:24:57.324999 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:57.324954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:24:57.327677 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:57.327648    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/628e1705-7c95-42b2-a3fd-c524a324c87a-metrics-tls\") pod \"dns-default-vnkct\" (UID: \"628e1705-7c95-42b2-a3fd-c524a324c87a\") " pod="openshift-dns/dns-default-vnkct"
Jul 01 19:24:57.580185 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:57.580102    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdgrk\""
Jul 01 19:24:57.587450 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:57.587409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vnkct"
Jul 01 19:24:57.927981 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:57.927943    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vnkct"]
Jul 01 19:24:57.931393 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:24:57.931360    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628e1705_7c95_42b2_a3fd_c524a324c87a.slice/crio-9dfd6e4ad1f866aca4c9960b34bec0d69923348e8b09366baf7715f3ef4361e2 WatchSource:0}: Error finding container 9dfd6e4ad1f866aca4c9960b34bec0d69923348e8b09366baf7715f3ef4361e2: Status 404 returned error can't find the container with id 9dfd6e4ad1f866aca4c9960b34bec0d69923348e8b09366baf7715f3ef4361e2
Jul 01 19:24:58.499873 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:24:58.499827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vnkct" event={"ID":"628e1705-7c95-42b2-a3fd-c524a324c87a","Type":"ContainerStarted","Data":"9dfd6e4ad1f866aca4c9960b34bec0d69923348e8b09366baf7715f3ef4361e2"}
Jul 01 19:25:00.508934 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:00.508876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vnkct" event={"ID":"628e1705-7c95-42b2-a3fd-c524a324c87a","Type":"ContainerStarted","Data":"7ed273e1f9e0dac633738d49bca816e2d65dbba6d3c2e0532bc5d87fdb0971dd"}
Jul 01 19:25:00.509350 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:00.508975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vnkct" event={"ID":"628e1705-7c95-42b2-a3fd-c524a324c87a","Type":"ContainerStarted","Data":"62e5c21b1ede508b2ef0d32cc9c8ded626cb9c9b5a5fc5762a2609ed91db287b"}
Jul 01 19:25:00.509350 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:00.509003    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vnkct"
Jul 01 19:25:00.528263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:00.528213    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vnkct" podStartSLOduration=129.397763276 podStartE2EDuration="2m11.528195976s" totalImagesPullingTime="2.1304327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:25:00.527426519 +0000 UTC m=+168.140469907" watchObservedRunningTime="2026-07-01 19:25:00.528195976 +0000 UTC m=+168.141239354"
Jul 01 19:25:08.537460 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:08.537425    2576 generic.go:356] "Generic (PLEG): container finished" podID="e2a85bfa-282d-4472-94ca-4811bb44335b" containerID="cd3a06d25603e9f03605cf50b9afe2360d3237b48841f394f0fe68a796b662b0" exitCode=0
Jul 01 19:25:08.537886 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:08.537479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw" event={"ID":"e2a85bfa-282d-4472-94ca-4811bb44335b","Type":"ContainerDied","Data":"cd3a06d25603e9f03605cf50b9afe2360d3237b48841f394f0fe68a796b662b0"}
Jul 01 19:25:08.537886 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:08.537745    2576 scope.go:117] "RemoveContainer" containerID="cd3a06d25603e9f03605cf50b9afe2360d3237b48841f394f0fe68a796b662b0"
Jul 01 19:25:09.542093 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:09.542062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-jjrkw" event={"ID":"e2a85bfa-282d-4472-94ca-4811bb44335b","Type":"ContainerStarted","Data":"51ab8205213ba5b2baa7ab81bbb5a36a112396efca9420f1407d1abbd67b9080"}
Jul 01 19:25:10.514558 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:25:10.514529    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vnkct"
Jul 01 19:27:12.740061 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:27:12.740025    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 19:28:03.427558 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.427521    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"]
Jul 01 19:28:03.430651 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.430634    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.433734 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.433708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 01 19:28:03.433863 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.433843    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s2hql\""
Jul 01 19:28:03.434887 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.434869    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 01 19:28:03.444261 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.444229    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"]
Jul 01 19:28:03.583295 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.583258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxb7\" (UniqueName: \"kubernetes.io/projected/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-kube-api-access-ftxb7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.583472 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.583312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.583472 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.583359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.684238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.684145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxb7\" (UniqueName: \"kubernetes.io/projected/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-kube-api-access-ftxb7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.684238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.684207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.684238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.684227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.684576 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.684559    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.684627 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.684597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.698847 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.698812    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxb7\" (UniqueName: \"kubernetes.io/projected/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-kube-api-access-ftxb7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.739950 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.739889    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:03.876318 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.876269    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"]
Jul 01 19:28:03.880203 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:28:03.880179    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329f1e27_727c_42ff_9aa1_d8df4a33cdf8.slice/crio-008bac716c984acb910020ae77df33256151a0f4de0747439e9c4d60548a2b28 WatchSource:0}: Error finding container 008bac716c984acb910020ae77df33256151a0f4de0747439e9c4d60548a2b28: Status 404 returned error can't find the container with id 008bac716c984acb910020ae77df33256151a0f4de0747439e9c4d60548a2b28
Jul 01 19:28:03.882413 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:03.882399    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:28:04.118890 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:04.118846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw" event={"ID":"329f1e27-727c-42ff-9aa1-d8df4a33cdf8","Type":"ContainerStarted","Data":"008bac716c984acb910020ae77df33256151a0f4de0747439e9c4d60548a2b28"}
Jul 01 19:28:13.154971 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:13.154934    2576 generic.go:356] "Generic (PLEG): container finished" podID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerID="c0262974533586ac0f88977436aa60270816514d52a421f1edcc518ad91b2b60" exitCode=0
Jul 01 19:28:13.155454 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:13.154986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw" event={"ID":"329f1e27-727c-42ff-9aa1-d8df4a33cdf8","Type":"ContainerDied","Data":"c0262974533586ac0f88977436aa60270816514d52a421f1edcc518ad91b2b60"}
Jul 01 19:28:18.173477 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:18.173439    2576 generic.go:356] "Generic (PLEG): container finished" podID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerID="74e0f981e4644ca241d27402bdca54b69fd11d11dd4141acaa5a6ffdb6859516" exitCode=0
Jul 01 19:28:18.173973 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:18.173534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw" event={"ID":"329f1e27-727c-42ff-9aa1-d8df4a33cdf8","Type":"ContainerDied","Data":"74e0f981e4644ca241d27402bdca54b69fd11d11dd4141acaa5a6ffdb6859516"}
Jul 01 19:28:26.206723 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:26.206691    2576 generic.go:356] "Generic (PLEG): container finished" podID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerID="85f3d751e65b1dc2d6232f2171e5dd925914c452cbcddbeed4386f080c8f5597" exitCode=0
Jul 01 19:28:26.207152 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:26.206756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw" event={"ID":"329f1e27-727c-42ff-9aa1-d8df4a33cdf8","Type":"ContainerDied","Data":"85f3d751e65b1dc2d6232f2171e5dd925914c452cbcddbeed4386f080c8f5597"}
Jul 01 19:28:27.364553 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.364515    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:27.372134 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.372110    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-bundle\") pod \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") "
Jul 01 19:28:27.372238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.372150    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftxb7\" (UniqueName: \"kubernetes.io/projected/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-kube-api-access-ftxb7\") pod \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") "
Jul 01 19:28:27.372238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.372193    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-util\") pod \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\" (UID: \"329f1e27-727c-42ff-9aa1-d8df4a33cdf8\") "
Jul 01 19:28:27.372807 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.372781    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-bundle" (OuterVolumeSpecName: "bundle") pod "329f1e27-727c-42ff-9aa1-d8df4a33cdf8" (UID: "329f1e27-727c-42ff-9aa1-d8df4a33cdf8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:28:27.374353 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.374334    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-kube-api-access-ftxb7" (OuterVolumeSpecName: "kube-api-access-ftxb7") pod "329f1e27-727c-42ff-9aa1-d8df4a33cdf8" (UID: "329f1e27-727c-42ff-9aa1-d8df4a33cdf8"). InnerVolumeSpecName "kube-api-access-ftxb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:28:27.377658 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.377633    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-util" (OuterVolumeSpecName: "util") pod "329f1e27-727c-42ff-9aa1-d8df4a33cdf8" (UID: "329f1e27-727c-42ff-9aa1-d8df4a33cdf8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:28:27.472686 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.472598    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-bundle\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:28:27.472686 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.472627    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxb7\" (UniqueName: \"kubernetes.io/projected/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-kube-api-access-ftxb7\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:28:27.472686 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:27.472637    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329f1e27-727c-42ff-9aa1-d8df4a33cdf8-util\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:28:28.214441 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:28.214407    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw"
Jul 01 19:28:28.214626 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:28.214402    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589mb2tw" event={"ID":"329f1e27-727c-42ff-9aa1-d8df4a33cdf8","Type":"ContainerDied","Data":"008bac716c984acb910020ae77df33256151a0f4de0747439e9c4d60548a2b28"}
Jul 01 19:28:28.214626 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:28.214531    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="008bac716c984acb910020ae77df33256151a0f4de0747439e9c4d60548a2b28"
Jul 01 19:28:35.985753 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.985714    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-k564c"]
Jul 01 19:28:35.986154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.985967    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerName="util"
Jul 01 19:28:35.986154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.985978    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerName="util"
Jul 01 19:28:35.986154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.985988    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerName="extract"
Jul 01 19:28:35.986154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.985993    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerName="extract"
Jul 01 19:28:35.986154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.986002    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerName="pull"
Jul 01 19:28:35.986154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.986009    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerName="pull"
Jul 01 19:28:35.986154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.986054    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="329f1e27-727c-42ff-9aa1-d8df4a33cdf8" containerName="extract"
Jul 01 19:28:35.988871 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:35.988855    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.003826 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.003800    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 19:28:36.004811 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.004796    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 01 19:28:36.005044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.005024    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 19:28:36.005044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.005038    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 19:28:36.005183 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.005057    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 19:28:36.005183 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.005064    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ks59s\""
Jul 01 19:28:36.011625 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.011603    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-k564c"]
Jul 01 19:28:36.026628 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.026600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.026783 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.026637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275j4\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-kube-api-access-275j4\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.026783 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.026666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ed37b7f-2b3e-4154-b06c-09a689284530-cabundle0\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.127480 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.127436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.127480 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.127483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275j4\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-kube-api-access-275j4\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.127746 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.127527    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ed37b7f-2b3e-4154-b06c-09a689284530-cabundle0\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.127746 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.127595    2576 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 01 19:28:36.127746 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.127627    2576 secret.go:281] references non-existent secret key: ca.crt
Jul 01 19:28:36.127746 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.127639    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 19:28:36.127746 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.127654    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-k564c: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 01 19:28:36.127746 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.127715    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates podName:5ed37b7f-2b3e-4154-b06c-09a689284530 nodeName:}" failed. No retries permitted until 2026-07-01 19:28:36.627696964 +0000 UTC m=+384.240740332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates") pod "keda-operator-68bff59c-k564c" (UID: "5ed37b7f-2b3e-4154-b06c-09a689284530") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 01 19:28:36.128285 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.128263    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ed37b7f-2b3e-4154-b06c-09a689284530-cabundle0\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.137086 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.137064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275j4\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-kube-api-access-275j4\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.306275 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.306243    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"]
Jul 01 19:28:36.309405 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.309382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.311801 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.311781    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 01 19:28:36.320175 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.320136    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"]
Jul 01 19:28:36.328817 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.328788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.328817 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.328819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/875ea546-ce37-4992-8795-91e719267b8e-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.329032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.328943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqsmf\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-kube-api-access-cqsmf\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.430054 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.430017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.430054 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.430063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/875ea546-ce37-4992-8795-91e719267b8e-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.430272 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.430116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqsmf\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-kube-api-access-cqsmf\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.430272 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.430164    2576 secret.go:281] references non-existent secret key: tls.crt
Jul 01 19:28:36.430272 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.430185    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 01 19:28:36.430272 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.430199    2576 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 01 19:28:36.430272 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.430217    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 01 19:28:36.430418 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.430281    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates podName:875ea546-ce37-4992-8795-91e719267b8e nodeName:}" failed. No retries permitted until 2026-07-01 19:28:36.930267697 +0000 UTC m=+384.543311062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates") pod "keda-metrics-apiserver-559f75f947-gdxh2" (UID: "875ea546-ce37-4992-8795-91e719267b8e") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 01 19:28:36.430552 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.430530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/875ea546-ce37-4992-8795-91e719267b8e-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.438813 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.438796    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqsmf\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-kube-api-access-cqsmf\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.631449 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.631346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:36.631604 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.631509    2576 secret.go:281] references non-existent secret key: ca.crt
Jul 01 19:28:36.631604 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.631531    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 19:28:36.631604 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.631539    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-k564c: references non-existent secret key: ca.crt
Jul 01 19:28:36.631604 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.631594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates podName:5ed37b7f-2b3e-4154-b06c-09a689284530 nodeName:}" failed. No retries permitted until 2026-07-01 19:28:37.631579323 +0000 UTC m=+385.244622687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates") pod "keda-operator-68bff59c-k564c" (UID: "5ed37b7f-2b3e-4154-b06c-09a689284530") : references non-existent secret key: ca.crt
Jul 01 19:28:36.933261 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:36.933157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:36.933427 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.933304    2576 secret.go:281] references non-existent secret key: tls.crt
Jul 01 19:28:36.933427 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.933320    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 01 19:28:36.933427 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.933337    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2: references non-existent secret key: tls.crt
Jul 01 19:28:36.933427 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:28:36.933396    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates podName:875ea546-ce37-4992-8795-91e719267b8e nodeName:}" failed. No retries permitted until 2026-07-01 19:28:37.933383226 +0000 UTC m=+385.546426594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates") pod "keda-metrics-apiserver-559f75f947-gdxh2" (UID: "875ea546-ce37-4992-8795-91e719267b8e") : references non-existent secret key: tls.crt
Jul 01 19:28:37.099639 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.099604    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-8b7zz"]
Jul 01 19:28:37.102708 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.102692    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.105233 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.105212    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 01 19:28:37.114031 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.114005    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-8b7zz"]
Jul 01 19:28:37.135416 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.135384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/06f1eaef-3817-497b-b5cd-54c7e70260c5-certificates\") pod \"keda-admission-5fbd5c4644-8b7zz\" (UID: \"06f1eaef-3817-497b-b5cd-54c7e70260c5\") " pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.135580 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.135425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7ls\" (UniqueName: \"kubernetes.io/projected/06f1eaef-3817-497b-b5cd-54c7e70260c5-kube-api-access-vb7ls\") pod \"keda-admission-5fbd5c4644-8b7zz\" (UID: \"06f1eaef-3817-497b-b5cd-54c7e70260c5\") " pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.236611 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.236519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/06f1eaef-3817-497b-b5cd-54c7e70260c5-certificates\") pod \"keda-admission-5fbd5c4644-8b7zz\" (UID: \"06f1eaef-3817-497b-b5cd-54c7e70260c5\") " pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.236611 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.236573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7ls\" (UniqueName: \"kubernetes.io/projected/06f1eaef-3817-497b-b5cd-54c7e70260c5-kube-api-access-vb7ls\") pod \"keda-admission-5fbd5c4644-8b7zz\" (UID: \"06f1eaef-3817-497b-b5cd-54c7e70260c5\") " pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.238999 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.238976    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/06f1eaef-3817-497b-b5cd-54c7e70260c5-certificates\") pod \"keda-admission-5fbd5c4644-8b7zz\" (UID: \"06f1eaef-3817-497b-b5cd-54c7e70260c5\") " pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.247242 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.247215    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7ls\" (UniqueName: \"kubernetes.io/projected/06f1eaef-3817-497b-b5cd-54c7e70260c5-kube-api-access-vb7ls\") pod \"keda-admission-5fbd5c4644-8b7zz\" (UID: \"06f1eaef-3817-497b-b5cd-54c7e70260c5\") " pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.412469 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.412426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:37.538986 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.538952    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-8b7zz"]
Jul 01 19:28:37.542232 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:28:37.542206    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f1eaef_3817_497b_b5cd_54c7e70260c5.slice/crio-5ac2ad0913bc58b829fbf29cd3dd37b37114017dbfca1ecee16af574eb1566ba WatchSource:0}: Error finding container 5ac2ad0913bc58b829fbf29cd3dd37b37114017dbfca1ecee16af574eb1566ba: Status 404 returned error can't find the container with id 5ac2ad0913bc58b829fbf29cd3dd37b37114017dbfca1ecee16af574eb1566ba
Jul 01 19:28:37.640488 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.640443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:37.642961 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.642894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ed37b7f-2b3e-4154-b06c-09a689284530-certificates\") pod \"keda-operator-68bff59c-k564c\" (UID: \"5ed37b7f-2b3e-4154-b06c-09a689284530\") " pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:37.797803 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.797760    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:37.927981 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.927950    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-k564c"]
Jul 01 19:28:37.931316 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:28:37.931282    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed37b7f_2b3e_4154_b06c_09a689284530.slice/crio-995454d2051a45bc75ee1dbf0cd05fdf4eb0ff630136ccde1bc27552cce78c61 WatchSource:0}: Error finding container 995454d2051a45bc75ee1dbf0cd05fdf4eb0ff630136ccde1bc27552cce78c61: Status 404 returned error can't find the container with id 995454d2051a45bc75ee1dbf0cd05fdf4eb0ff630136ccde1bc27552cce78c61
Jul 01 19:28:37.943639 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.943611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:37.946082 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:37.946056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/875ea546-ce37-4992-8795-91e719267b8e-certificates\") pod \"keda-metrics-apiserver-559f75f947-gdxh2\" (UID: \"875ea546-ce37-4992-8795-91e719267b8e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:38.120119 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:38.120019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:38.249323 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:38.249297    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"]
Jul 01 19:28:38.252762 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:28:38.252737    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875ea546_ce37_4992_8795_91e719267b8e.slice/crio-18f35320c3ced0da5ed15e02a44864c184637eb12b5e5d1fb6b5bb6110397374 WatchSource:0}: Error finding container 18f35320c3ced0da5ed15e02a44864c184637eb12b5e5d1fb6b5bb6110397374: Status 404 returned error can't find the container with id 18f35320c3ced0da5ed15e02a44864c184637eb12b5e5d1fb6b5bb6110397374
Jul 01 19:28:38.253079 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:38.253052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-k564c" event={"ID":"5ed37b7f-2b3e-4154-b06c-09a689284530","Type":"ContainerStarted","Data":"995454d2051a45bc75ee1dbf0cd05fdf4eb0ff630136ccde1bc27552cce78c61"}
Jul 01 19:28:38.254324 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:38.254302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz" event={"ID":"06f1eaef-3817-497b-b5cd-54c7e70260c5","Type":"ContainerStarted","Data":"5ac2ad0913bc58b829fbf29cd3dd37b37114017dbfca1ecee16af574eb1566ba"}
Jul 01 19:28:39.262235 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:39.262195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2" event={"ID":"875ea546-ce37-4992-8795-91e719267b8e","Type":"ContainerStarted","Data":"18f35320c3ced0da5ed15e02a44864c184637eb12b5e5d1fb6b5bb6110397374"}
Jul 01 19:28:40.268545 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:40.268503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz" event={"ID":"06f1eaef-3817-497b-b5cd-54c7e70260c5","Type":"ContainerStarted","Data":"1f237a87df937b8bed3f52bd360306a66d37c89585c26216170e6b11aade4e12"}
Jul 01 19:28:40.269065 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:40.268709    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:28:40.291934 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:40.291861    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz" podStartSLOduration=0.904965828 podStartE2EDuration="3.291837871s" totalImagesPullingTime="2.386872043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:40.290355781 +0000 UTC m=+387.903399171" watchObservedRunningTime="2026-07-01 19:28:40.291837871 +0000 UTC m=+387.904881260"
Jul 01 19:28:42.276982 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:42.276947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2" event={"ID":"875ea546-ce37-4992-8795-91e719267b8e","Type":"ContainerStarted","Data":"3aa7ac18b21176243ec775c3549aa279febc929cb9ca49b6985135cc69ba99a9"}
Jul 01 19:28:42.277409 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:42.277025    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:28:42.300427 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:42.300364    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2" podStartSLOduration=3.270920606 podStartE2EDuration="6.300342449s" totalImagesPullingTime="3.029421843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:42.298746141 +0000 UTC m=+389.911789529" watchObservedRunningTime="2026-07-01 19:28:42.300342449 +0000 UTC m=+389.913385837"
Jul 01 19:28:44.286062 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:44.286020    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-k564c" event={"ID":"5ed37b7f-2b3e-4154-b06c-09a689284530","Type":"ContainerStarted","Data":"03a8b0114a0a8d72358f8065832c2df3efcf7f73bda52d2658c4760bf10e5e44"}
Jul 01 19:28:44.286481 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:44.286149    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:28:44.306207 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:44.306150    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-k564c" podStartSLOduration=3.872934324 podStartE2EDuration="9.306132129s" totalImagesPullingTime="5.433197805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:44.305856708 +0000 UTC m=+391.918900096" watchObservedRunningTime="2026-07-01 19:28:44.306132129 +0000 UTC m=+391.919175519"
Jul 01 19:28:53.286176 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:28:53.286145    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-gdxh2"
Jul 01 19:29:01.274411 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:29:01.274374    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-8b7zz"
Jul 01 19:29:05.292133 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:29:05.292095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-k564c"
Jul 01 19:30:57.432074 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.432046    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-q6v7c"]
Jul 01 19:30:57.435032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.435016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:57.437565 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.437544    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 19:30:57.437660 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.437595    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 19:30:57.437660 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.437629    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-64z7f\""
Jul 01 19:30:57.438819 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.438803    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 19:30:57.444616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.444593    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-q6v7c"]
Jul 01 19:30:57.557972 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.557939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p8v\" (UniqueName: \"kubernetes.io/projected/c62add65-1abd-4907-826b-dd66dfa99917-kube-api-access-v4p8v\") pod \"odh-model-controller-84b54b7f97-q6v7c\" (UID: \"c62add65-1abd-4907-826b-dd66dfa99917\") " pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:57.557972 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.557978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c62add65-1abd-4907-826b-dd66dfa99917-cert\") pod \"odh-model-controller-84b54b7f97-q6v7c\" (UID: \"c62add65-1abd-4907-826b-dd66dfa99917\") " pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:57.658622 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.658588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p8v\" (UniqueName: \"kubernetes.io/projected/c62add65-1abd-4907-826b-dd66dfa99917-kube-api-access-v4p8v\") pod \"odh-model-controller-84b54b7f97-q6v7c\" (UID: \"c62add65-1abd-4907-826b-dd66dfa99917\") " pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:57.658622 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.658625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c62add65-1abd-4907-826b-dd66dfa99917-cert\") pod \"odh-model-controller-84b54b7f97-q6v7c\" (UID: \"c62add65-1abd-4907-826b-dd66dfa99917\") " pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:57.658887 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:30:57.658726    2576 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 01 19:30:57.658887 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:30:57.658789    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c62add65-1abd-4907-826b-dd66dfa99917-cert podName:c62add65-1abd-4907-826b-dd66dfa99917 nodeName:}" failed. No retries permitted until 2026-07-01 19:30:58.158774008 +0000 UTC m=+525.771817373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c62add65-1abd-4907-826b-dd66dfa99917-cert") pod "odh-model-controller-84b54b7f97-q6v7c" (UID: "c62add65-1abd-4907-826b-dd66dfa99917") : secret "odh-model-controller-webhook-cert" not found
Jul 01 19:30:57.670707 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:57.670685    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p8v\" (UniqueName: \"kubernetes.io/projected/c62add65-1abd-4907-826b-dd66dfa99917-kube-api-access-v4p8v\") pod \"odh-model-controller-84b54b7f97-q6v7c\" (UID: \"c62add65-1abd-4907-826b-dd66dfa99917\") " pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:58.162568 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:58.162536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c62add65-1abd-4907-826b-dd66dfa99917-cert\") pod \"odh-model-controller-84b54b7f97-q6v7c\" (UID: \"c62add65-1abd-4907-826b-dd66dfa99917\") " pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:58.165018 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:58.164995    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c62add65-1abd-4907-826b-dd66dfa99917-cert\") pod \"odh-model-controller-84b54b7f97-q6v7c\" (UID: \"c62add65-1abd-4907-826b-dd66dfa99917\") " pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:58.344678 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:58.344630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:30:58.472719 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:58.472695    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-q6v7c"]
Jul 01 19:30:58.475326 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:30:58.475295    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62add65_1abd_4907_826b_dd66dfa99917.slice/crio-3fb444e566b3efcc0750f609dad7c7e40d353b5312df4d28b55c97ad5a84f535 WatchSource:0}: Error finding container 3fb444e566b3efcc0750f609dad7c7e40d353b5312df4d28b55c97ad5a84f535: Status 404 returned error can't find the container with id 3fb444e566b3efcc0750f609dad7c7e40d353b5312df4d28b55c97ad5a84f535
Jul 01 19:30:58.759833 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:30:58.759804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-q6v7c" event={"ID":"c62add65-1abd-4907-826b-dd66dfa99917","Type":"ContainerStarted","Data":"3fb444e566b3efcc0750f609dad7c7e40d353b5312df4d28b55c97ad5a84f535"}
Jul 01 19:31:02.778453 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:31:02.778417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-q6v7c" event={"ID":"c62add65-1abd-4907-826b-dd66dfa99917","Type":"ContainerStarted","Data":"cdce4bf674b7b0c3089d01d0069bf95dd0cab8a92f4aeb7e863c27a9bb68f61f"}
Jul 01 19:31:02.778861 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:31:02.778539    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:31:02.798235 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:31:02.798186    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-q6v7c" podStartSLOduration=2.240147939 podStartE2EDuration="5.798169715s" totalImagesPullingTime="3.558021776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:31:02.797274558 +0000 UTC m=+530.410317944" watchObservedRunningTime="2026-07-01 19:31:02.798169715 +0000 UTC m=+530.411213099"
Jul 01 19:31:13.784725 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:31:13.784644    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-q6v7c"
Jul 01 19:32:45.863805 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.863727    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"]
Jul 01 19:32:45.866981 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.866962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:45.869930 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.869893    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:32:45.869930 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.869903    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b8262-kube-rbac-proxy-sar-config\""
Jul 01 19:32:45.870097 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.869969    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:32:45.870097 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.869907    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:32:45.871188 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.871167    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bqdbq\""
Jul 01 19:32:45.871284 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.871187    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:32:45.871284 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.871188    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b8262-predictor-serving-cert\""
Jul 01 19:32:45.881543 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.881524    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"]
Jul 01 19:32:45.977647 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.977612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:45.977810 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.977655    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:45.977810 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.977708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:45.977810 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.977738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:45.977810 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:45.977795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrt9\" (UniqueName: \"kubernetes.io/projected/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-api-access-nxrt9\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.078553 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.078515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.078553 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.078557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.078766 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.078583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.078766 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.078621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.078766 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.078666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrt9\" (UniqueName: \"kubernetes.io/projected/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-api-access-nxrt9\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.083943 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.079266    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.083943 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.079654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.083943 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.079709    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.083943 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.082087    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.094138 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.094109    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrt9\" (UniqueName: \"kubernetes.io/projected/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-api-access-nxrt9\") pod \"isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.178131 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.178036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:32:46.314466 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:46.314338    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"]
Jul 01 19:32:46.317109 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:32:46.317077    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d1328f_7a58_4c14_a76a_2ad9eea2e04b.slice/crio-ac7923da49c150a96ecf6bce2dc3062b2c09c1e96c56a370b221dc75e9bda33a WatchSource:0}: Error finding container ac7923da49c150a96ecf6bce2dc3062b2c09c1e96c56a370b221dc75e9bda33a: Status 404 returned error can't find the container with id ac7923da49c150a96ecf6bce2dc3062b2c09c1e96c56a370b221dc75e9bda33a
Jul 01 19:32:47.130668 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:47.130625    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerStarted","Data":"ac7923da49c150a96ecf6bce2dc3062b2c09c1e96c56a370b221dc75e9bda33a"}
Jul 01 19:32:50.145931 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:50.145828    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerStarted","Data":"46aee82027446d80f5ceaa29564af4870c803b867335908e8e3132dcd83657bf"}
Jul 01 19:32:54.162297 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:54.162260    2576 generic.go:356] "Generic (PLEG): container finished" podID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerID="46aee82027446d80f5ceaa29564af4870c803b867335908e8e3132dcd83657bf" exitCode=0
Jul 01 19:32:54.162297 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:32:54.162301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerDied","Data":"46aee82027446d80f5ceaa29564af4870c803b867335908e8e3132dcd83657bf"}
Jul 01 19:33:08.681642 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:08.681621    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:33:09.233079 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:09.233044    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerStarted","Data":"25783cb6a3acf0aa739cca03786c120a7984957e6d2ffd125126648bbe4cd40c"}
Jul 01 19:33:12.248488 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:12.248452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerStarted","Data":"b86ec8df62c4988601085a187352dd5c2fbddd87151c0456102944dcf8c21d5b"}
Jul 01 19:33:14.259123 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:14.259087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerStarted","Data":"1c8d77a90b42341b7b7270209c51d837213f911844e4f60ebe66f074263cfa87"}
Jul 01 19:33:14.259544 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:14.259315    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:33:14.259544 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:14.259364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:33:14.260652 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:14.260613    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:14.287487 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:14.287424    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podStartSLOduration=2.546672574 podStartE2EDuration="29.287402691s" totalImagesPullingTime="23.143808345s" totalInitContainerRuntime="3.596921772s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:45 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:33:14.285188172 +0000 UTC m=+661.898231561" watchObservedRunningTime="2026-07-01 19:33:14.287402691 +0000 UTC m=+661.900446079"
Jul 01 19:33:15.263158 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:15.263121    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:33:15.263632 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:15.263225    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:15.264275 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:15.264242    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:16.267244 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:16.267203    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:16.267685 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:16.267611    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:16.271053 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:16.271032    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:33:17.271018 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:17.270977    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:17.271505 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:17.271267    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:27.271697 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:27.271642    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:27.272107 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:27.272047    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:37.271867 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:37.271817    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:37.272403 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:37.272380    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:47.271579 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:47.271527    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:47.272054 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:47.272033    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:57.271613 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:57.271572    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:33:57.272130 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:33:57.272016    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:07.271270 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:07.271170    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:34:07.271633 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:07.271592    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:17.271143 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:17.271099    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:34:17.271540 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:17.271481    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:27.271685 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:27.271651    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:34:27.272214 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:27.272188    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:34:41.128263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.128226    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"]
Jul 01 19:34:41.131799 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.131740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.132619 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.132599    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"]
Jul 01 19:34:41.132826 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.132800    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" containerID="cri-o://1c8d77a90b42341b7b7270209c51d837213f911844e4f60ebe66f074263cfa87" gracePeriod=30
Jul 01 19:34:41.132883 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.132821    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" containerID="cri-o://25783cb6a3acf0aa739cca03786c120a7984957e6d2ffd125126648bbe4cd40c" gracePeriod=30
Jul 01 19:34:41.132959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.132882    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" containerID="cri-o://b86ec8df62c4988601085a187352dd5c2fbddd87151c0456102944dcf8c21d5b" gracePeriod=30
Jul 01 19:34:41.134212 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.134183    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-54928-predictor-serving-cert\""
Jul 01 19:34:41.134212 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.134211    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-54928-kube-rbac-proxy-sar-config\""
Jul 01 19:34:41.145435 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.145405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"]
Jul 01 19:34:41.197242 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.197210    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"]
Jul 01 19:34:41.200815 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.200790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.203767 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.203744    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-54928-kube-rbac-proxy-sar-config\""
Jul 01 19:34:41.203879 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.203744    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-54928-predictor-serving-cert\""
Jul 01 19:34:41.211631 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.211610    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"]
Jul 01 19:34:41.267349 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.267308    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 19:34:41.314782 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.314743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d9c4cf7-69a6-4754-8733-65530de70476-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.314782 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.314781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.315081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.314804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-cabundle-cert\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.315081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.314849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.315081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.314888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072ba039-eaf9-4716-94c8-3064036dac27-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.315081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.314980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxm5\" (UniqueName: \"kubernetes.io/projected/7d9c4cf7-69a6-4754-8733-65530de70476-kube-api-access-npxm5\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.315081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.315013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-cabundle-cert\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.315081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.315041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d9c4cf7-69a6-4754-8733-65530de70476-proxy-tls\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.315293 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.315085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9wc\" (UniqueName: \"kubernetes.io/projected/072ba039-eaf9-4716-94c8-3064036dac27-kube-api-access-jr9wc\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.315293 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.315117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072ba039-eaf9-4716-94c8-3064036dac27-proxy-tls\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.416221 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d9c4cf7-69a6-4754-8733-65530de70476-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.416221 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.416221 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-cabundle-cert\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.416508 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.416508 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072ba039-eaf9-4716-94c8-3064036dac27-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.416508 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxm5\" (UniqueName: \"kubernetes.io/projected/7d9c4cf7-69a6-4754-8733-65530de70476-kube-api-access-npxm5\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.416508 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-cabundle-cert\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.416508 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d9c4cf7-69a6-4754-8733-65530de70476-proxy-tls\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.416508 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9wc\" (UniqueName: \"kubernetes.io/projected/072ba039-eaf9-4716-94c8-3064036dac27-kube-api-access-jr9wc\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.416508 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072ba039-eaf9-4716-94c8-3064036dac27-proxy-tls\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.416827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d9c4cf7-69a6-4754-8733-65530de70476-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.416827 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416711    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072ba039-eaf9-4716-94c8-3064036dac27-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.416995 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.416970    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-cabundle-cert\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.417069 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.417018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.417337 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.417315    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.417423 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.417392    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-cabundle-cert\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.419027 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.419009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072ba039-eaf9-4716-94c8-3064036dac27-proxy-tls\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.419146 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.419043    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d9c4cf7-69a6-4754-8733-65530de70476-proxy-tls\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.424529 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.424502    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9wc\" (UniqueName: \"kubernetes.io/projected/072ba039-eaf9-4716-94c8-3064036dac27-kube-api-access-jr9wc\") pod \"isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.425269 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.425247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxm5\" (UniqueName: \"kubernetes.io/projected/7d9c4cf7-69a6-4754-8733-65530de70476-kube-api-access-npxm5\") pod \"isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.452546 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.452523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:41.511331 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.511297    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:34:41.577972 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.577944    2576 generic.go:356] "Generic (PLEG): container finished" podID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerID="b86ec8df62c4988601085a187352dd5c2fbddd87151c0456102944dcf8c21d5b" exitCode=2
Jul 01 19:34:41.578093 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.578023    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerDied","Data":"b86ec8df62c4988601085a187352dd5c2fbddd87151c0456102944dcf8c21d5b"}
Jul 01 19:34:41.588810 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.588788    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"]
Jul 01 19:34:41.591480 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:34:41.591453    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072ba039_eaf9_4716_94c8_3064036dac27.slice/crio-e3cde655c7b555862406bedb22dee05f633f62f1d7f98966542a990db5e0620d WatchSource:0}: Error finding container e3cde655c7b555862406bedb22dee05f633f62f1d7f98966542a990db5e0620d: Status 404 returned error can't find the container with id e3cde655c7b555862406bedb22dee05f633f62f1d7f98966542a990db5e0620d
Jul 01 19:34:41.651235 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:41.651178    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"]
Jul 01 19:34:41.653980 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:34:41.653949    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9c4cf7_69a6_4754_8733_65530de70476.slice/crio-304ed6d587f0b736d342514537c753bdfc3bf69d3554e744aea96aedd985f0be WatchSource:0}: Error finding container 304ed6d587f0b736d342514537c753bdfc3bf69d3554e744aea96aedd985f0be: Status 404 returned error can't find the container with id 304ed6d587f0b736d342514537c753bdfc3bf69d3554e744aea96aedd985f0be
Jul 01 19:34:42.582965 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:42.582932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerStarted","Data":"0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595"}
Jul 01 19:34:42.583391 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:42.582992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerStarted","Data":"304ed6d587f0b736d342514537c753bdfc3bf69d3554e744aea96aedd985f0be"}
Jul 01 19:34:42.584388 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:42.584352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerStarted","Data":"d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9"}
Jul 01 19:34:42.584506 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:42.584400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerStarted","Data":"e3cde655c7b555862406bedb22dee05f633f62f1d7f98966542a990db5e0620d"}
Jul 01 19:34:46.268175 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:46.268133    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 19:34:46.602264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:46.602184    2576 generic.go:356] "Generic (PLEG): container finished" podID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerID="25783cb6a3acf0aa739cca03786c120a7984957e6d2ffd125126648bbe4cd40c" exitCode=0
Jul 01 19:34:46.602414 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:46.602256    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerDied","Data":"25783cb6a3acf0aa739cca03786c120a7984957e6d2ffd125126648bbe4cd40c"}
Jul 01 19:34:46.603525 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:46.603496    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d9c4cf7-69a6-4754-8733-65530de70476" containerID="0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595" exitCode=0
Jul 01 19:34:46.603650 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:46.603569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerDied","Data":"0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595"}
Jul 01 19:34:46.604981 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:46.604960    2576 generic.go:356] "Generic (PLEG): container finished" podID="072ba039-eaf9-4716-94c8-3064036dac27" containerID="d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9" exitCode=0
Jul 01 19:34:46.605065 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:46.605005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerDied","Data":"d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9"}
Jul 01 19:34:47.271586 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.271542    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:47.273282 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.273248    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:34:47.612267 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.612170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerStarted","Data":"62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46"}
Jul 01 19:34:47.612267 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.612263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerStarted","Data":"58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5"}
Jul 01 19:34:47.612642 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.612623    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:47.612787 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.612768    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:47.614190 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.614149    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:34:47.633836 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:47.633775    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podStartSLOduration=2.699118513 podStartE2EDuration="6.633752586s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.934634073s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:34:47.631324144 +0000 UTC m=+755.244367532" watchObservedRunningTime="2026-07-01 19:34:47.633752586 +0000 UTC m=+755.246795974"
Jul 01 19:34:48.617793 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:48.617751    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:34:51.267733 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:51.267687    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 19:34:51.268249 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:51.267872    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:34:53.624024 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:53.623991    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:34:53.624720 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:53.624690    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:34:56.267776 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:56.267735    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 19:34:57.271903 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:57.271860    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:57.272389 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:34:57.272219    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:35:01.267750 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:01.267696    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 19:35:03.624978 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:03.624937    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:35:04.687276 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:04.687241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerStarted","Data":"d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c"}
Jul 01 19:35:05.693574 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:05.693538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerStarted","Data":"7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257"}
Jul 01 19:35:05.694037 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:05.693695    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:35:05.714840 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:05.714790    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podStartSLOduration=2.7948722740000003 podStartE2EDuration="24.714775699s" totalImagesPullingTime="17.996535821s" totalInitContainerRuntime="3.923367604s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:35:05.712333345 +0000 UTC m=+773.325376734" watchObservedRunningTime="2026-07-01 19:35:05.714775699 +0000 UTC m=+773.327819136"
Jul 01 19:35:06.268091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:06.268051    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 19:35:06.698261 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:06.698173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:35:06.699337 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:06.699310    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:35:07.271794 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:07.271747    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:35:07.272020 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:07.271937    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:35:07.272223 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:07.272201    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 01 19:35:07.272304 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:07.272293    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:35:07.702562 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:07.702477    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:35:11.268133 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.268091    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 01 19:35:11.722904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.722868    2576 generic.go:356] "Generic (PLEG): container finished" podID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerID="1c8d77a90b42341b7b7270209c51d837213f911844e4f60ebe66f074263cfa87" exitCode=0
Jul 01 19:35:11.722904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.722889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerDied","Data":"1c8d77a90b42341b7b7270209c51d837213f911844e4f60ebe66f074263cfa87"}
Jul 01 19:35:11.818985 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.818951    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:35:11.883540 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.883495    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-proxy-tls\") pod \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") "
Jul 01 19:35:11.883765 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.883561    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-cabundle-cert\") pod \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") "
Jul 01 19:35:11.883765 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.883585    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxrt9\" (UniqueName: \"kubernetes.io/projected/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-api-access-nxrt9\") pod \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") "
Jul 01 19:35:11.883765 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.883631    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kserve-provision-location\") pod \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") "
Jul 01 19:35:11.883765 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.883660    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-rbac-proxy-sar-config\") pod \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\" (UID: \"82d1328f-7a58-4c14-a76a-2ad9eea2e04b\") "
Jul 01 19:35:11.883993 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.883911    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82d1328f-7a58-4c14-a76a-2ad9eea2e04b" (UID: "82d1328f-7a58-4c14-a76a-2ad9eea2e04b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:35:11.884052 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.884015    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82d1328f-7a58-4c14-a76a-2ad9eea2e04b" (UID: "82d1328f-7a58-4c14-a76a-2ad9eea2e04b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:35:11.884109 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.884052    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82d1328f-7a58-4c14-a76a-2ad9eea2e04b" (UID: "82d1328f-7a58-4c14-a76a-2ad9eea2e04b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:35:11.885806 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.885784    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82d1328f-7a58-4c14-a76a-2ad9eea2e04b" (UID: "82d1328f-7a58-4c14-a76a-2ad9eea2e04b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:35:11.886006 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.885854    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-api-access-nxrt9" (OuterVolumeSpecName: "kube-api-access-nxrt9") pod "82d1328f-7a58-4c14-a76a-2ad9eea2e04b" (UID: "82d1328f-7a58-4c14-a76a-2ad9eea2e04b"). InnerVolumeSpecName "kube-api-access-nxrt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:35:11.984709 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.984636    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:35:11.984709 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.984665    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:35:11.984709 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.984674    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxrt9\" (UniqueName: \"kubernetes.io/projected/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-api-access-nxrt9\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:35:11.984709 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.984682    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:35:11.984709 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:11.984690    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82d1328f-7a58-4c14-a76a-2ad9eea2e04b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:35:12.707488 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.707460    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:35:12.708047 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.708016    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:35:12.731718 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.731688    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn" event={"ID":"82d1328f-7a58-4c14-a76a-2ad9eea2e04b","Type":"ContainerDied","Data":"ac7923da49c150a96ecf6bce2dc3062b2c09c1e96c56a370b221dc75e9bda33a"}
Jul 01 19:35:12.731718 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.731709    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"
Jul 01 19:35:12.731944 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.731754    2576 scope.go:117] "RemoveContainer" containerID="1c8d77a90b42341b7b7270209c51d837213f911844e4f60ebe66f074263cfa87"
Jul 01 19:35:12.741554 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.741521    2576 scope.go:117] "RemoveContainer" containerID="b86ec8df62c4988601085a187352dd5c2fbddd87151c0456102944dcf8c21d5b"
Jul 01 19:35:12.749004 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.748985    2576 scope.go:117] "RemoveContainer" containerID="25783cb6a3acf0aa739cca03786c120a7984957e6d2ffd125126648bbe4cd40c"
Jul 01 19:35:12.756735 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.756708    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"]
Jul 01 19:35:12.757175 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.757138    2576 scope.go:117] "RemoveContainer" containerID="46aee82027446d80f5ceaa29564af4870c803b867335908e8e3132dcd83657bf"
Jul 01 19:35:12.758780 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.758763    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn"]
Jul 01 19:35:12.794379 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:12.794353    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" path="/var/lib/kubelet/pods/82d1328f-7a58-4c14-a76a-2ad9eea2e04b/volumes"
Jul 01 19:35:13.624832 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:13.624798    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:35:22.708593 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:22.708554    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:35:23.624579 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:23.624540    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:35:32.707937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:32.707877    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:35:33.624672 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:33.624627    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:35:42.708808 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:42.708711    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:35:43.624910 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:43.624869    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:35:52.708057 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:52.708013    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:35:53.625133 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:35:53.625096    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:36:02.708480 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:02.708438    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:36:03.625600 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:03.625570    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:36:12.709083 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:12.709056    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:36:31.443103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.443071    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"]
Jul 01 19:36:31.443528 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.443307    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" containerID="cri-o://58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5" gracePeriod=30
Jul 01 19:36:31.443528 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.443387    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kube-rbac-proxy" containerID="cri-o://62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46" gracePeriod=30
Jul 01 19:36:31.450149 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450126    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"]
Jul 01 19:36:31.450411 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450400    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent"
Jul 01 19:36:31.450453 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450413    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent"
Jul 01 19:36:31.450453 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450431    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container"
Jul 01 19:36:31.450453 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450437    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container"
Jul 01 19:36:31.450453 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450443    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy"
Jul 01 19:36:31.450453 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450448    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy"
Jul 01 19:36:31.450601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="storage-initializer"
Jul 01 19:36:31.450601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450466    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="storage-initializer"
Jul 01 19:36:31.450601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450510    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kserve-container"
Jul 01 19:36:31.450601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450518    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="kube-rbac-proxy"
Jul 01 19:36:31.450601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.450523    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d1328f-7a58-4c14-a76a-2ad9eea2e04b" containerName="agent"
Jul 01 19:36:31.453568 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.453550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.456286 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.456268    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-31388-predictor-serving-cert\""
Jul 01 19:36:31.456364 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.456289    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-31388-kube-rbac-proxy-sar-config\""
Jul 01 19:36:31.468911 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.468893    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"]
Jul 01 19:36:31.513542 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.513512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7vj\" (UniqueName: \"kubernetes.io/projected/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-api-access-dm7vj\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.513542 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.513546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.513787 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.513609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.513787 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.513633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.513787 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.513656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.591133 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.591105    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"]
Jul 01 19:36:31.591364 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.591329    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" containerID="cri-o://d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c" gracePeriod=30
Jul 01 19:36:31.591422 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.591367    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kube-rbac-proxy" containerID="cri-o://7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257" gracePeriod=30
Jul 01 19:36:31.614584 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.614559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7vj\" (UniqueName: \"kubernetes.io/projected/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-api-access-dm7vj\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.614688 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.614590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.614756 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.614737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.614816 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.614774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.614966 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.614902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.615079 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:31.615052    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-31388-predictor-serving-cert" not found
Jul 01 19:36:31.615207 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:31.615130    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls podName:e13bfad3-7f49-412a-8d4d-f5cf92addc44 nodeName:}" failed. No retries permitted until 2026-07-01 19:36:32.115107176 +0000 UTC m=+859.728150543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" (UID: "e13bfad3-7f49-412a-8d4d-f5cf92addc44") : secret "isvc-sklearn-graph-raw-hpa-31388-predictor-serving-cert" not found
Jul 01 19:36:31.615207 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.615170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.615477 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.615460    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.615571 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.615541    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:31.624304 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:31.624284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7vj\" (UniqueName: \"kubernetes.io/projected/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-api-access-dm7vj\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:32.024856 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.024825    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d9c4cf7-69a6-4754-8733-65530de70476" containerID="7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257" exitCode=2
Jul 01 19:36:32.025058 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.024897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerDied","Data":"7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257"}
Jul 01 19:36:32.026885 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.026865    2576 generic.go:356] "Generic (PLEG): container finished" podID="072ba039-eaf9-4716-94c8-3064036dac27" containerID="62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46" exitCode=2
Jul 01 19:36:32.026988 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.026892    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerDied","Data":"62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46"}
Jul 01 19:36:32.118631 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.118601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:32.120891 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.120868    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:32.369732 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.369650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:32.499173 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.499148    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"]
Jul 01 19:36:32.501314 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:36:32.501282    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13bfad3_7f49_412a_8d4d_f5cf92addc44.slice/crio-18242bd43e1c53afa001a927ffc0c621c0e0d1dcee83f267abe1a9b20adc090a WatchSource:0}: Error finding container 18242bd43e1c53afa001a927ffc0c621c0e0d1dcee83f267abe1a9b20adc090a: Status 404 returned error can't find the container with id 18242bd43e1c53afa001a927ffc0c621c0e0d1dcee83f267abe1a9b20adc090a
Jul 01 19:36:32.703388 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.703283    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 19:36:32.708874 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:32.708839    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:36:33.032544 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:33.032511    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerStarted","Data":"a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9"}
Jul 01 19:36:33.032743 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:33.032585    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerStarted","Data":"18242bd43e1c53afa001a927ffc0c621c0e0d1dcee83f267abe1a9b20adc090a"}
Jul 01 19:36:33.618118 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:33.618071    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 19:36:33.625078 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:33.625048    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 19:36:35.340122 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.340094    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:36:35.445888 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.445796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d9c4cf7-69a6-4754-8733-65530de70476-kserve-provision-location\") pod \"7d9c4cf7-69a6-4754-8733-65530de70476\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") "
Jul 01 19:36:35.445888 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.445853    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d9c4cf7-69a6-4754-8733-65530de70476-proxy-tls\") pod \"7d9c4cf7-69a6-4754-8733-65530de70476\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") "
Jul 01 19:36:35.445888 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.445872    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-kube-rbac-proxy-sar-config\") pod \"7d9c4cf7-69a6-4754-8733-65530de70476\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") "
Jul 01 19:36:35.446197 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.445908    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npxm5\" (UniqueName: \"kubernetes.io/projected/7d9c4cf7-69a6-4754-8733-65530de70476-kube-api-access-npxm5\") pod \"7d9c4cf7-69a6-4754-8733-65530de70476\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") "
Jul 01 19:36:35.446197 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.445948    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-cabundle-cert\") pod \"7d9c4cf7-69a6-4754-8733-65530de70476\" (UID: \"7d9c4cf7-69a6-4754-8733-65530de70476\") "
Jul 01 19:36:35.446299 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.446202    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9c4cf7-69a6-4754-8733-65530de70476-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d9c4cf7-69a6-4754-8733-65530de70476" (UID: "7d9c4cf7-69a6-4754-8733-65530de70476"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:36:35.446406 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.446379    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d9c4cf7-69a6-4754-8733-65530de70476" (UID: "7d9c4cf7-69a6-4754-8733-65530de70476"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:35.446406 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.446395    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d9c4cf7-69a6-4754-8733-65530de70476" (UID: "7d9c4cf7-69a6-4754-8733-65530de70476"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:35.447995 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.447974    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9c4cf7-69a6-4754-8733-65530de70476-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d9c4cf7-69a6-4754-8733-65530de70476" (UID: "7d9c4cf7-69a6-4754-8733-65530de70476"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:36:35.448118 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.448097    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9c4cf7-69a6-4754-8733-65530de70476-kube-api-access-npxm5" (OuterVolumeSpecName: "kube-api-access-npxm5") pod "7d9c4cf7-69a6-4754-8733-65530de70476" (UID: "7d9c4cf7-69a6-4754-8733-65530de70476"). InnerVolumeSpecName "kube-api-access-npxm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:36:35.547158 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.547126    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d9c4cf7-69a6-4754-8733-65530de70476-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:35.547158 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.547153    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d9c4cf7-69a6-4754-8733-65530de70476-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:35.547158 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.547163    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:35.547394 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.547172    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxm5\" (UniqueName: \"kubernetes.io/projected/7d9c4cf7-69a6-4754-8733-65530de70476-kube-api-access-npxm5\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:35.547394 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:35.547180    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d9c4cf7-69a6-4754-8733-65530de70476-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:36.046625 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.046593    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d9c4cf7-69a6-4754-8733-65530de70476" containerID="d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c" exitCode=0
Jul 01 19:36:36.046735 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.046663    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"
Jul 01 19:36:36.046735 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.046679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerDied","Data":"d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c"}
Jul 01 19:36:36.046839 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.046757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz" event={"ID":"7d9c4cf7-69a6-4754-8733-65530de70476","Type":"ContainerDied","Data":"304ed6d587f0b736d342514537c753bdfc3bf69d3554e744aea96aedd985f0be"}
Jul 01 19:36:36.046839 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.046788    2576 scope.go:117] "RemoveContainer" containerID="7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257"
Jul 01 19:36:36.092071 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.092049    2576 scope.go:117] "RemoveContainer" containerID="d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c"
Jul 01 19:36:36.100190 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.100166    2576 scope.go:117] "RemoveContainer" containerID="0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595"
Jul 01 19:36:36.105688 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.105662    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"]
Jul 01 19:36:36.108090 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.108071    2576 scope.go:117] "RemoveContainer" containerID="7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257"
Jul 01 19:36:36.108373 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:36.108353    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257\": container with ID starting with 7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257 not found: ID does not exist" containerID="7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257"
Jul 01 19:36:36.108420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.108382    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257"} err="failed to get container status \"7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257\": rpc error: code = NotFound desc = could not find container \"7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257\": container with ID starting with 7b5cdbe2e8ea8c060c2b80cadeac905b55b55113c69cf9696ab56b9309791257 not found: ID does not exist"
Jul 01 19:36:36.108420 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.108415    2576 scope.go:117] "RemoveContainer" containerID="d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c"
Jul 01 19:36:36.108658 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:36.108639    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c\": container with ID starting with d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c not found: ID does not exist" containerID="d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c"
Jul 01 19:36:36.108698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.108668    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c"} err="failed to get container status \"d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c\": rpc error: code = NotFound desc = could not find container \"d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c\": container with ID starting with d6acc71195e5f6229b6834aad30958e2220aa7ae803329d536ab4a699730472c not found: ID does not exist"
Jul 01 19:36:36.108698 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.108690    2576 scope.go:117] "RemoveContainer" containerID="0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595"
Jul 01 19:36:36.108990 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:36.108959    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595\": container with ID starting with 0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595 not found: ID does not exist" containerID="0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595"
Jul 01 19:36:36.109079 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.108999    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595"} err="failed to get container status \"0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595\": rpc error: code = NotFound desc = could not find container \"0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595\": container with ID starting with 0ca7758ed2e15c64b05574875f24c4fd731b16dc55db7573273eb5879abd4595 not found: ID does not exist"
Jul 01 19:36:36.111359 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.111329    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz"]
Jul 01 19:36:36.198796 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.198775    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:36:36.251806 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.251782    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-cabundle-cert\") pod \"072ba039-eaf9-4716-94c8-3064036dac27\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") "
Jul 01 19:36:36.251947 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.251818    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072ba039-eaf9-4716-94c8-3064036dac27-proxy-tls\") pod \"072ba039-eaf9-4716-94c8-3064036dac27\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") "
Jul 01 19:36:36.251947 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.251858    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072ba039-eaf9-4716-94c8-3064036dac27-kserve-provision-location\") pod \"072ba039-eaf9-4716-94c8-3064036dac27\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") "
Jul 01 19:36:36.251947 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.251878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr9wc\" (UniqueName: \"kubernetes.io/projected/072ba039-eaf9-4716-94c8-3064036dac27-kube-api-access-jr9wc\") pod \"072ba039-eaf9-4716-94c8-3064036dac27\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") "
Jul 01 19:36:36.251947 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.251903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-kube-rbac-proxy-sar-config\") pod \"072ba039-eaf9-4716-94c8-3064036dac27\" (UID: \"072ba039-eaf9-4716-94c8-3064036dac27\") "
Jul 01 19:36:36.252263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.252154    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "072ba039-eaf9-4716-94c8-3064036dac27" (UID: "072ba039-eaf9-4716-94c8-3064036dac27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:36.252354 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.252214    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072ba039-eaf9-4716-94c8-3064036dac27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "072ba039-eaf9-4716-94c8-3064036dac27" (UID: "072ba039-eaf9-4716-94c8-3064036dac27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:36:36.252354 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.252271    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "072ba039-eaf9-4716-94c8-3064036dac27" (UID: "072ba039-eaf9-4716-94c8-3064036dac27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:36.253894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.253864    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072ba039-eaf9-4716-94c8-3064036dac27-kube-api-access-jr9wc" (OuterVolumeSpecName: "kube-api-access-jr9wc") pod "072ba039-eaf9-4716-94c8-3064036dac27" (UID: "072ba039-eaf9-4716-94c8-3064036dac27"). InnerVolumeSpecName "kube-api-access-jr9wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:36:36.254011 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.253961    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072ba039-eaf9-4716-94c8-3064036dac27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "072ba039-eaf9-4716-94c8-3064036dac27" (UID: "072ba039-eaf9-4716-94c8-3064036dac27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:36:36.353463 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.353383    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:36.353463 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.353409    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/072ba039-eaf9-4716-94c8-3064036dac27-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:36.353463 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.353418    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072ba039-eaf9-4716-94c8-3064036dac27-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:36.353463 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.353427    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr9wc\" (UniqueName: \"kubernetes.io/projected/072ba039-eaf9-4716-94c8-3064036dac27-kube-api-access-jr9wc\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:36.353463 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.353436    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/072ba039-eaf9-4716-94c8-3064036dac27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:36:36.794222 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:36.794186    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" path="/var/lib/kubelet/pods/7d9c4cf7-69a6-4754-8733-65530de70476/volumes"
Jul 01 19:36:37.052376 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.052299    2576 generic.go:356] "Generic (PLEG): container finished" podID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerID="a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9" exitCode=0
Jul 01 19:36:37.052516 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.052374    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerDied","Data":"a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9"}
Jul 01 19:36:37.054214 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.054193    2576 generic.go:356] "Generic (PLEG): container finished" podID="072ba039-eaf9-4716-94c8-3064036dac27" containerID="58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5" exitCode=0
Jul 01 19:36:37.054344 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.054223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerDied","Data":"58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5"}
Jul 01 19:36:37.054344 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.054243    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"
Jul 01 19:36:37.054344 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.054279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7" event={"ID":"072ba039-eaf9-4716-94c8-3064036dac27","Type":"ContainerDied","Data":"e3cde655c7b555862406bedb22dee05f633f62f1d7f98966542a990db5e0620d"}
Jul 01 19:36:37.054344 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.054316    2576 scope.go:117] "RemoveContainer" containerID="62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46"
Jul 01 19:36:37.062095 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.062074    2576 scope.go:117] "RemoveContainer" containerID="58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5"
Jul 01 19:36:37.069166 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.069146    2576 scope.go:117] "RemoveContainer" containerID="d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9"
Jul 01 19:36:37.076558 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.076544    2576 scope.go:117] "RemoveContainer" containerID="62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46"
Jul 01 19:36:37.076796 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:37.076780    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46\": container with ID starting with 62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46 not found: ID does not exist" containerID="62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46"
Jul 01 19:36:37.076844 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.076803    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46"} err="failed to get container status \"62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46\": rpc error: code = NotFound desc = could not find container \"62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46\": container with ID starting with 62646ff630c31fe98eec6cb9d654ae1777eafff6ea1e9962611a02003d6fcb46 not found: ID does not exist"
Jul 01 19:36:37.076844 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.076819    2576 scope.go:117] "RemoveContainer" containerID="58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5"
Jul 01 19:36:37.077067 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:37.077050    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5\": container with ID starting with 58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5 not found: ID does not exist" containerID="58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5"
Jul 01 19:36:37.077117 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.077072    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5"} err="failed to get container status \"58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5\": rpc error: code = NotFound desc = could not find container \"58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5\": container with ID starting with 58177c232ea197ff11c82d4726ca9ee11dc13e92c6d53d4de8b67365e689c6c5 not found: ID does not exist"
Jul 01 19:36:37.077117 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.077086    2576 scope.go:117] "RemoveContainer" containerID="d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9"
Jul 01 19:36:37.077322 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:36:37.077308    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9\": container with ID starting with d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9 not found: ID does not exist" containerID="d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9"
Jul 01 19:36:37.077368 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.077325    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9"} err="failed to get container status \"d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9\": rpc error: code = NotFound desc = could not find container \"d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9\": container with ID starting with d459d188718c4122ee6a55c9f47b05459946f378f41c0d87965309d8bca566a9 not found: ID does not exist"
Jul 01 19:36:37.083996 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.083954    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"]
Jul 01 19:36:37.087370 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:37.087351    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7"]
Jul 01 19:36:38.065432 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:38.065400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerStarted","Data":"53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92"}
Jul 01 19:36:38.065823 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:38.065482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerStarted","Data":"ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236"}
Jul 01 19:36:38.065823 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:38.065791    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:38.065959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:38.065932    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:38.067258 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:38.067235    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:36:38.087880 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:38.087828    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podStartSLOduration=3.452061743 podStartE2EDuration="7.08781444s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.635752697s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:36:38.08575244 +0000 UTC m=+865.698795827" watchObservedRunningTime="2026-07-01 19:36:38.08781444 +0000 UTC m=+865.700857877"
Jul 01 19:36:38.794669 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:38.794634    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072ba039-eaf9-4716-94c8-3064036dac27" path="/var/lib/kubelet/pods/072ba039-eaf9-4716-94c8-3064036dac27/volumes"
Jul 01 19:36:39.069334 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:39.069238    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:36:40.072822 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:40.072778    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:36:45.077903 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:45.077874    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:36:45.078455 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:45.078425    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:36:55.078829 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:36:55.078789    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:37:05.079238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:37:05.079158    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:37:15.078564 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:37:15.078522    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:37:25.078477 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:37:25.078434    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:37:35.079267 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:37:35.079223    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:37:45.078930 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:37:45.078879    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:37:55.079985 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:37:55.079955    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:38:21.684824 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.684792    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"]
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685102    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685118    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685135    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="storage-initializer"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685144    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="storage-initializer"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685161    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kube-rbac-proxy"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685169    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kube-rbac-proxy"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685178    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kube-rbac-proxy"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685186    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kube-rbac-proxy"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685205    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="storage-initializer"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685214    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="storage-initializer"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685225    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685234    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container"
Jul 01 19:38:21.685311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685313    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kserve-container"
Jul 01 19:38:21.685728 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685325    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kserve-container"
Jul 01 19:38:21.685728 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685332    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="072ba039-eaf9-4716-94c8-3064036dac27" containerName="kube-rbac-proxy"
Jul 01 19:38:21.685728 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.685338    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9c4cf7-69a6-4754-8733-65530de70476" containerName="kube-rbac-proxy"
Jul 01 19:38:21.688260 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.688244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.691518 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.691489    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9f9d5-predictor-serving-cert\""
Jul 01 19:38:21.691879 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.691769    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9f9d5-kube-rbac-proxy-sar-config\""
Jul 01 19:38:21.702931 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.702899    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"]
Jul 01 19:38:21.722831 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.722804    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"]
Jul 01 19:38:21.723061 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.723027    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" containerID="cri-o://ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236" gracePeriod=30
Jul 01 19:38:21.723107 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.723053    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kube-rbac-proxy" containerID="cri-o://53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92" gracePeriod=30
Jul 01 19:38:21.729459 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.729251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.729459 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.729294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5923fb80-b30c-41c3-981a-553eb825e384-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.729459 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.729366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwg7\" (UniqueName: \"kubernetes.io/projected/5923fb80-b30c-41c3-981a-553eb825e384-kube-api-access-2bwg7\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.830479 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.830442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.830632 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.830503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5923fb80-b30c-41c3-981a-553eb825e384-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.830632 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:38:21.830563    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-serving-cert: secret "message-dumper-raw-9f9d5-predictor-serving-cert" not found
Jul 01 19:38:21.830741 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:38:21.830636    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls podName:5923fb80-b30c-41c3-981a-553eb825e384 nodeName:}" failed. No retries permitted until 2026-07-01 19:38:22.3306184 +0000 UTC m=+969.943661765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls") pod "message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" (UID: "5923fb80-b30c-41c3-981a-553eb825e384") : secret "message-dumper-raw-9f9d5-predictor-serving-cert" not found
Jul 01 19:38:21.830741 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.830569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwg7\" (UniqueName: \"kubernetes.io/projected/5923fb80-b30c-41c3-981a-553eb825e384-kube-api-access-2bwg7\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.831127 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.831110    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5923fb80-b30c-41c3-981a-553eb825e384-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:21.841688 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:21.841666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwg7\" (UniqueName: \"kubernetes.io/projected/5923fb80-b30c-41c3-981a-553eb825e384-kube-api-access-2bwg7\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:22.335630 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:22.335595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:22.338012 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:22.337990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls\") pod \"message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") " pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:22.449786 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:22.449752    2576 generic.go:356] "Generic (PLEG): container finished" podID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerID="53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92" exitCode=2
Jul 01 19:38:22.449960 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:22.449823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerDied","Data":"53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92"}
Jul 01 19:38:22.597623 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:22.597515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:22.727103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:22.727023    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"]
Jul 01 19:38:22.729560 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:38:22.729530    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5923fb80_b30c_41c3_981a_553eb825e384.slice/crio-e6d0b01fa2a0a730ea336402320e84b2796ff4a8b3f3cc5ae51b962d36c29243 WatchSource:0}: Error finding container e6d0b01fa2a0a730ea336402320e84b2796ff4a8b3f3cc5ae51b962d36c29243: Status 404 returned error can't find the container with id e6d0b01fa2a0a730ea336402320e84b2796ff4a8b3f3cc5ae51b962d36c29243
Jul 01 19:38:22.731652 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:22.731634    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:38:23.455183 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:23.455148    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" event={"ID":"5923fb80-b30c-41c3-981a-553eb825e384","Type":"ContainerStarted","Data":"e6d0b01fa2a0a730ea336402320e84b2796ff4a8b3f3cc5ae51b962d36c29243"}
Jul 01 19:38:24.460776 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:24.460744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" event={"ID":"5923fb80-b30c-41c3-981a-553eb825e384","Type":"ContainerStarted","Data":"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"}
Jul 01 19:38:24.461208 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:24.460807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" event={"ID":"5923fb80-b30c-41c3-981a-553eb825e384","Type":"ContainerStarted","Data":"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"}
Jul 01 19:38:24.461208 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:24.460832    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:24.461208 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:24.460877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:24.463070 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:24.463045    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:24.480789 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:24.480744    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" podStartSLOduration=2.212492871 podStartE2EDuration="3.480730145s" totalImagesPullingTime="1.268237274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:38:24.478758133 +0000 UTC m=+972.091801521" watchObservedRunningTime="2026-07-01 19:38:24.480730145 +0000 UTC m=+972.093773531"
Jul 01 19:38:25.073616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:25.073556    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 19:38:25.078884 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:25.078861    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 19:38:26.468633 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.468588    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:38:26.471691 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.471666    2576 generic.go:356] "Generic (PLEG): container finished" podID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerID="ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236" exitCode=0
Jul 01 19:38:26.471826 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.471718    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"
Jul 01 19:38:26.471826 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.471748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerDied","Data":"ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236"}
Jul 01 19:38:26.471826 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.471822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7" event={"ID":"e13bfad3-7f49-412a-8d4d-f5cf92addc44","Type":"ContainerDied","Data":"18242bd43e1c53afa001a927ffc0c621c0e0d1dcee83f267abe1a9b20adc090a"}
Jul 01 19:38:26.472008 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.471861    2576 scope.go:117] "RemoveContainer" containerID="53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92"
Jul 01 19:38:26.479192 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.479173    2576 scope.go:117] "RemoveContainer" containerID="ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236"
Jul 01 19:38:26.485938 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.485890    2576 scope.go:117] "RemoveContainer" containerID="a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9"
Jul 01 19:38:26.493107 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.493085    2576 scope.go:117] "RemoveContainer" containerID="53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92"
Jul 01 19:38:26.493449 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:38:26.493427    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92\": container with ID starting with 53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92 not found: ID does not exist" containerID="53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92"
Jul 01 19:38:26.493538 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.493456    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92"} err="failed to get container status \"53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92\": rpc error: code = NotFound desc = could not find container \"53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92\": container with ID starting with 53f4d9e7b77952daa1223513b7f7fca4498c984e1601bec23b61309cb782da92 not found: ID does not exist"
Jul 01 19:38:26.493538 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.493474    2576 scope.go:117] "RemoveContainer" containerID="ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236"
Jul 01 19:38:26.493756 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:38:26.493728    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236\": container with ID starting with ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236 not found: ID does not exist" containerID="ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236"
Jul 01 19:38:26.493846 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.493761    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236"} err="failed to get container status \"ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236\": rpc error: code = NotFound desc = could not find container \"ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236\": container with ID starting with ee3286cb6a604cad3e04d65869abc6ddfce8f4d9092b33d833012859b1aa3236 not found: ID does not exist"
Jul 01 19:38:26.493846 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.493782    2576 scope.go:117] "RemoveContainer" containerID="a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9"
Jul 01 19:38:26.494135 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:38:26.494107    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9\": container with ID starting with a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9 not found: ID does not exist" containerID="a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9"
Jul 01 19:38:26.494228 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.494141    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9"} err="failed to get container status \"a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9\": rpc error: code = NotFound desc = could not find container \"a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9\": container with ID starting with a75420b09f76a21bb10f878c5b121e463b16df8119c1bffa8ac188d20f19b8f9 not found: ID does not exist"
Jul 01 19:38:26.569757 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.569717    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-rbac-proxy-sar-config\") pod \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") "
Jul 01 19:38:26.569757 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.569751    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-cabundle-cert\") pod \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") "
Jul 01 19:38:26.569991 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.569796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls\") pod \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") "
Jul 01 19:38:26.569991 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.569830    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kserve-provision-location\") pod \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") "
Jul 01 19:38:26.569991 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.569866    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm7vj\" (UniqueName: \"kubernetes.io/projected/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-api-access-dm7vj\") pod \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\" (UID: \"e13bfad3-7f49-412a-8d4d-f5cf92addc44\") "
Jul 01 19:38:26.570231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.570182    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e13bfad3-7f49-412a-8d4d-f5cf92addc44" (UID: "e13bfad3-7f49-412a-8d4d-f5cf92addc44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:38:26.570231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.570199    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e13bfad3-7f49-412a-8d4d-f5cf92addc44" (UID: "e13bfad3-7f49-412a-8d4d-f5cf92addc44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:38:26.570231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.570206    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e13bfad3-7f49-412a-8d4d-f5cf92addc44" (UID: "e13bfad3-7f49-412a-8d4d-f5cf92addc44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:38:26.571959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.571938    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e13bfad3-7f49-412a-8d4d-f5cf92addc44" (UID: "e13bfad3-7f49-412a-8d4d-f5cf92addc44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:38:26.572194 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.572175    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-api-access-dm7vj" (OuterVolumeSpecName: "kube-api-access-dm7vj") pod "e13bfad3-7f49-412a-8d4d-f5cf92addc44" (UID: "e13bfad3-7f49-412a-8d4d-f5cf92addc44"). InnerVolumeSpecName "kube-api-access-dm7vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:38:26.671437 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.671403    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e13bfad3-7f49-412a-8d4d-f5cf92addc44-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:38:26.671437 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.671432    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:38:26.671437 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.671445    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7vj\" (UniqueName: \"kubernetes.io/projected/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-api-access-dm7vj\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:38:26.671655 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.671460    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:38:26.671655 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.671473    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e13bfad3-7f49-412a-8d4d-f5cf92addc44-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:38:26.795350 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.795320    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"]
Jul 01 19:38:26.795825 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:26.795808    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7"]
Jul 01 19:38:28.793784 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:28.793752    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" path="/var/lib/kubelet/pods/e13bfad3-7f49-412a-8d4d-f5cf92addc44/volumes"
Jul 01 19:38:31.476766 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:31.476739    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:38:41.713546 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713460    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"]
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713754    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="storage-initializer"
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713764    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="storage-initializer"
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713774    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container"
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713779    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container"
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713785    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kube-rbac-proxy"
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713791    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kube-rbac-proxy"
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713840    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kube-rbac-proxy"
Jul 01 19:38:41.713902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.713851    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13bfad3-7f49-412a-8d4d-f5cf92addc44" containerName="kserve-container"
Jul 01 19:38:41.716876 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.716859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.719359 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.719339    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:38:41.719481 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.719463    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9f9d5-predictor-serving-cert\""
Jul 01 19:38:41.719556 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.719470    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:38:41.719556 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.719470    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9f9d5-kube-rbac-proxy-sar-config\""
Jul 01 19:38:41.729272 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.729246    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"]
Jul 01 19:38:41.791383 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.791353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.791383 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.791386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01661e93-9bfc-40d1-a6ae-984f76c7d516-proxy-tls\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.791595 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.791406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xcg\" (UniqueName: \"kubernetes.io/projected/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-api-access-v9xcg\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.791595 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.791454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01661e93-9bfc-40d1-a6ae-984f76c7d516-kserve-provision-location\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.791595 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.791494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-cabundle-cert\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.892682 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.892647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-cabundle-cert\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.892860 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.892694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.892860 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.892714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01661e93-9bfc-40d1-a6ae-984f76c7d516-proxy-tls\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.892860 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.892813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xcg\" (UniqueName: \"kubernetes.io/projected/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-api-access-v9xcg\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.893070 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.892867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01661e93-9bfc-40d1-a6ae-984f76c7d516-kserve-provision-location\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.893389 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.893364    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-cabundle-cert\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.893511 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.893415    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01661e93-9bfc-40d1-a6ae-984f76c7d516-kserve-provision-location\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.893558 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.893504    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.895294 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.895273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01661e93-9bfc-40d1-a6ae-984f76c7d516-proxy-tls\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:41.901472 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:41.901442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xcg\" (UniqueName: \"kubernetes.io/projected/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-api-access-v9xcg\") pod \"isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:42.026758 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:42.026731    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:42.162390 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:42.162366    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"]
Jul 01 19:38:42.164057 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:38:42.164029    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01661e93_9bfc_40d1_a6ae_984f76c7d516.slice/crio-37139ef65faae1062cbad0152b756112d5cd925091f62b282d96302794fb39ba WatchSource:0}: Error finding container 37139ef65faae1062cbad0152b756112d5cd925091f62b282d96302794fb39ba: Status 404 returned error can't find the container with id 37139ef65faae1062cbad0152b756112d5cd925091f62b282d96302794fb39ba
Jul 01 19:38:42.531754 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:42.531717    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerStarted","Data":"85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2"}
Jul 01 19:38:42.531937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:42.531826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerStarted","Data":"37139ef65faae1062cbad0152b756112d5cd925091f62b282d96302794fb39ba"}
Jul 01 19:38:46.546886 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:46.546856    2576 generic.go:356] "Generic (PLEG): container finished" podID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerID="85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2" exitCode=0
Jul 01 19:38:46.547284 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:46.546955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerDied","Data":"85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2"}
Jul 01 19:38:47.557674 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:47.557636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerStarted","Data":"31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622"}
Jul 01 19:38:47.558156 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:47.557743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerStarted","Data":"4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5"}
Jul 01 19:38:47.558156 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:47.557786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerStarted","Data":"ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1"}
Jul 01 19:38:47.558156 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:47.558073    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:47.581560 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:47.581491    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podStartSLOduration=3.175410756 podStartE2EDuration="6.581472937s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.406062181s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:38:47.579582972 +0000 UTC m=+995.192626360" watchObservedRunningTime="2026-07-01 19:38:47.581472937 +0000 UTC m=+995.194516324"
Jul 01 19:38:48.561864 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:48.561833    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:48.562249 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:48.561907    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:48.563452 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:48.563423    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:38:48.564136 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:48.564114    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:38:49.565866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:49.565819    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:38:49.566344 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:49.566317    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:38:54.570805 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:54.570777    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:38:54.571484 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:54.571455    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:38:54.571682 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:38:54.571661    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:39:04.571742 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:04.571699    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:39:04.572219 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:04.572196    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:39:14.572145 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:14.572105    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:39:14.572659 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:14.572633    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:39:24.571858 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:24.571812    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:39:24.572442 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:24.572413    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:39:34.572140 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:34.572091    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:39:34.572628 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:34.572463    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:39:44.571347 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:44.571303    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:39:44.571903 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:44.571849    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:39:54.571748 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:54.571705    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:39:54.572254 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:39:54.572147    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:40:04.572551 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:04.572472    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:40:04.572964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:04.572941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:40:16.715340 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:16.715303    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4_5923fb80-b30c-41c3-981a-553eb825e384/kserve-container/0.log"
Jul 01 19:40:17.007589 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.007559    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"]
Jul 01 19:40:17.010877 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.010857    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.011816 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.011783    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"]
Jul 01 19:40:17.012167 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.012073    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" containerID="cri-o://31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622" gracePeriod=30
Jul 01 19:40:17.012167 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.012070    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" containerID="cri-o://ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1" gracePeriod=30
Jul 01 19:40:17.012167 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.012094    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy" containerID="cri-o://4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5" gracePeriod=30
Jul 01 19:40:17.014005 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.013985    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5a351-predictor-serving-cert\""
Jul 01 19:40:17.014311 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.014291    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5a351-kube-rbac-proxy-sar-config\""
Jul 01 19:40:17.031999 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.031977    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"]
Jul 01 19:40:17.099873 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.099840    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"]
Jul 01 19:40:17.100093 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.100063    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kserve-container" containerID="cri-o://c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64" gracePeriod=30
Jul 01 19:40:17.100243 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.100099    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kube-rbac-proxy" containerID="cri-o://a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359" gracePeriod=30
Jul 01 19:40:17.140221 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.140191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324ad3ad-1875-4ac1-af68-d8048aef08df-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.140352 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.140224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqzpl\" (UniqueName: \"kubernetes.io/projected/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-api-access-xqzpl\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.140352 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.140254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324ad3ad-1875-4ac1-af68-d8048aef08df-proxy-tls\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.140352 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.140307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.140506 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.140375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.241804 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.241774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324ad3ad-1875-4ac1-af68-d8048aef08df-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.241999 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.241811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpl\" (UniqueName: \"kubernetes.io/projected/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-api-access-xqzpl\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.241999 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.241842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324ad3ad-1875-4ac1-af68-d8048aef08df-proxy-tls\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.241999 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.241865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.241999 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.241887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.242280 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.242255    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324ad3ad-1875-4ac1-af68-d8048aef08df-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.242576 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.242545    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.242675 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.242610    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.244724 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.244695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324ad3ad-1875-4ac1-af68-d8048aef08df-proxy-tls\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.255160 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.255137    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqzpl\" (UniqueName: \"kubernetes.io/projected/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-api-access-xqzpl\") pod \"isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.328911 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.328881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:17.341107 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.341084    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:40:17.443539 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.443498    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5923fb80-b30c-41c3-981a-553eb825e384-kube-rbac-proxy-sar-config\") pod \"5923fb80-b30c-41c3-981a-553eb825e384\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") "
Jul 01 19:40:17.443708 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.443678    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bwg7\" (UniqueName: \"kubernetes.io/projected/5923fb80-b30c-41c3-981a-553eb825e384-kube-api-access-2bwg7\") pod \"5923fb80-b30c-41c3-981a-553eb825e384\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") "
Jul 01 19:40:17.443779 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.443717    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls\") pod \"5923fb80-b30c-41c3-981a-553eb825e384\" (UID: \"5923fb80-b30c-41c3-981a-553eb825e384\") "
Jul 01 19:40:17.443894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.443870    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5923fb80-b30c-41c3-981a-553eb825e384-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5923fb80-b30c-41c3-981a-553eb825e384" (UID: "5923fb80-b30c-41c3-981a-553eb825e384"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:40:17.444016 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.444001    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5923fb80-b30c-41c3-981a-553eb825e384-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:17.445788 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.445764    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5923fb80-b30c-41c3-981a-553eb825e384" (UID: "5923fb80-b30c-41c3-981a-553eb825e384"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:40:17.445890 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.445768    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5923fb80-b30c-41c3-981a-553eb825e384-kube-api-access-2bwg7" (OuterVolumeSpecName: "kube-api-access-2bwg7") pod "5923fb80-b30c-41c3-981a-553eb825e384" (UID: "5923fb80-b30c-41c3-981a-553eb825e384"). InnerVolumeSpecName "kube-api-access-2bwg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:40:17.464430 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.464408    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"]
Jul 01 19:40:17.466837 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:40:17.466818    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324ad3ad_1875_4ac1_af68_d8048aef08df.slice/crio-7e3ee86f4d276b5fbef6fbf88c7ce576dae126865df6ffb4471eec21e9dac139 WatchSource:0}: Error finding container 7e3ee86f4d276b5fbef6fbf88c7ce576dae126865df6ffb4471eec21e9dac139: Status 404 returned error can't find the container with id 7e3ee86f4d276b5fbef6fbf88c7ce576dae126865df6ffb4471eec21e9dac139
Jul 01 19:40:17.544837 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.544782    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwg7\" (UniqueName: \"kubernetes.io/projected/5923fb80-b30c-41c3-981a-553eb825e384-kube-api-access-2bwg7\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:17.544837 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.544806    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5923fb80-b30c-41c3-981a-553eb825e384-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:17.887065 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.886977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerStarted","Data":"ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8"}
Jul 01 19:40:17.887065 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.887058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerStarted","Data":"7e3ee86f4d276b5fbef6fbf88c7ce576dae126865df6ffb4471eec21e9dac139"}
Jul 01 19:40:17.889485 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.889461    2576 generic.go:356] "Generic (PLEG): container finished" podID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerID="4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5" exitCode=2
Jul 01 19:40:17.889600 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.889497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerDied","Data":"4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5"}
Jul 01 19:40:17.891070 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.891049    2576 generic.go:356] "Generic (PLEG): container finished" podID="5923fb80-b30c-41c3-981a-553eb825e384" containerID="a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359" exitCode=2
Jul 01 19:40:17.891070 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.891069    2576 generic.go:356] "Generic (PLEG): container finished" podID="5923fb80-b30c-41c3-981a-553eb825e384" containerID="c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64" exitCode=2
Jul 01 19:40:17.891178 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.891082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" event={"ID":"5923fb80-b30c-41c3-981a-553eb825e384","Type":"ContainerDied","Data":"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"}
Jul 01 19:40:17.891178 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.891127    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"
Jul 01 19:40:17.891178 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.891145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" event={"ID":"5923fb80-b30c-41c3-981a-553eb825e384","Type":"ContainerDied","Data":"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"}
Jul 01 19:40:17.891297 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.891179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4" event={"ID":"5923fb80-b30c-41c3-981a-553eb825e384","Type":"ContainerDied","Data":"e6d0b01fa2a0a730ea336402320e84b2796ff4a8b3f3cc5ae51b962d36c29243"}
Jul 01 19:40:17.891297 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.891210    2576 scope.go:117] "RemoveContainer" containerID="a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"
Jul 01 19:40:17.899057 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.899039    2576 scope.go:117] "RemoveContainer" containerID="c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"
Jul 01 19:40:17.907323 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.907300    2576 scope.go:117] "RemoveContainer" containerID="a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"
Jul 01 19:40:17.907654 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:40:17.907629    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359\": container with ID starting with a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359 not found: ID does not exist" containerID="a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"
Jul 01 19:40:17.907743 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.907663    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"} err="failed to get container status \"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359\": rpc error: code = NotFound desc = could not find container \"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359\": container with ID starting with a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359 not found: ID does not exist"
Jul 01 19:40:17.907743 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.907685    2576 scope.go:117] "RemoveContainer" containerID="c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"
Jul 01 19:40:17.907970 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:40:17.907953    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64\": container with ID starting with c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64 not found: ID does not exist" containerID="c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"
Jul 01 19:40:17.908017 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.907978    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"} err="failed to get container status \"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64\": rpc error: code = NotFound desc = could not find container \"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64\": container with ID starting with c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64 not found: ID does not exist"
Jul 01 19:40:17.908017 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.907999    2576 scope.go:117] "RemoveContainer" containerID="a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"
Jul 01 19:40:17.908257 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.908236    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359"} err="failed to get container status \"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359\": rpc error: code = NotFound desc = could not find container \"a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359\": container with ID starting with a5044da7bc725fadc44dc8acaa8e8c70f28a712b4024ddae19ede2994927b359 not found: ID does not exist"
Jul 01 19:40:17.908257 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.908255    2576 scope.go:117] "RemoveContainer" containerID="c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"
Jul 01 19:40:17.908515 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.908496    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64"} err="failed to get container status \"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64\": rpc error: code = NotFound desc = could not find container \"c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64\": container with ID starting with c33c9255ec630e92e55beed88a0bddb172192bf5d5106db0bc7e53c6b993fb64 not found: ID does not exist"
Jul 01 19:40:17.919551 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.919506    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"]
Jul 01 19:40:17.925879 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:17.925856    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4"]
Jul 01 19:40:18.794373 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:18.794339    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5923fb80-b30c-41c3-981a-553eb825e384" path="/var/lib/kubelet/pods/5923fb80-b30c-41c3-981a-553eb825e384/volumes"
Jul 01 19:40:19.566569 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:19.566523    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:40:21.907997 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:21.907964    2576 generic.go:356] "Generic (PLEG): container finished" podID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerID="ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8" exitCode=0
Jul 01 19:40:21.908363 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:21.908010    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerDied","Data":"ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8"}
Jul 01 19:40:21.910489 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:21.910468    2576 generic.go:356] "Generic (PLEG): container finished" podID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerID="ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1" exitCode=0
Jul 01 19:40:21.910601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:21.910509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerDied","Data":"ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1"}
Jul 01 19:40:22.916661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:22.916631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerStarted","Data":"1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c"}
Jul 01 19:40:22.917090 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:22.916687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerStarted","Data":"d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b"}
Jul 01 19:40:22.917090 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:22.916970    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:22.917090 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:22.917028    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:22.918319 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:22.918294    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:22.940617 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:22.940569    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podStartSLOduration=3.413332178 podStartE2EDuration="6.940554885s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.527222707s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:40:22.939134067 +0000 UTC m=+1090.552177455" watchObservedRunningTime="2026-07-01 19:40:22.940554885 +0000 UTC m=+1090.553598294"
Jul 01 19:40:23.920753 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:23.920711    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:24.566315 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:24.566273    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:40:24.572091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:24.572053    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:40:24.573238 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:24.573214    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:40:28.925487 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:28.925457    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:40:28.926140 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:28.926113    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:29.566586 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:29.566544    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:40:29.566767 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:29.566670    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:40:34.566532 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:34.566486    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:40:34.572220 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:34.572186    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:40:34.573651 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:34.573630    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:40:38.926410 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:38.926366    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:39.566483 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:39.566441    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:40:44.566527 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:44.566480    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:40:44.572212 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:44.572182    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:40:44.572321 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:44.572310    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:40:44.573388 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:44.573365    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:40:44.573483 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:44.573469    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:40:47.195403 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.195369    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:40:47.258666 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.258630    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-cabundle-cert\") pod \"01661e93-9bfc-40d1-a6ae-984f76c7d516\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") "
Jul 01 19:40:47.258821 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.258678    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01661e93-9bfc-40d1-a6ae-984f76c7d516-proxy-tls\") pod \"01661e93-9bfc-40d1-a6ae-984f76c7d516\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") "
Jul 01 19:40:47.258821 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.258724    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01661e93-9bfc-40d1-a6ae-984f76c7d516-kserve-provision-location\") pod \"01661e93-9bfc-40d1-a6ae-984f76c7d516\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") "
Jul 01 19:40:47.258821 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.258778    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9xcg\" (UniqueName: \"kubernetes.io/projected/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-api-access-v9xcg\") pod \"01661e93-9bfc-40d1-a6ae-984f76c7d516\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") "
Jul 01 19:40:47.258821 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.258819    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-rbac-proxy-sar-config\") pod \"01661e93-9bfc-40d1-a6ae-984f76c7d516\" (UID: \"01661e93-9bfc-40d1-a6ae-984f76c7d516\") "
Jul 01 19:40:47.259061 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.259038    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01661e93-9bfc-40d1-a6ae-984f76c7d516" (UID: "01661e93-9bfc-40d1-a6ae-984f76c7d516"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:40:47.259100 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.259076    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01661e93-9bfc-40d1-a6ae-984f76c7d516-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01661e93-9bfc-40d1-a6ae-984f76c7d516" (UID: "01661e93-9bfc-40d1-a6ae-984f76c7d516"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:40:47.259223 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.259201    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01661e93-9bfc-40d1-a6ae-984f76c7d516" (UID: "01661e93-9bfc-40d1-a6ae-984f76c7d516"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:40:47.260830 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.260811    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01661e93-9bfc-40d1-a6ae-984f76c7d516-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01661e93-9bfc-40d1-a6ae-984f76c7d516" (UID: "01661e93-9bfc-40d1-a6ae-984f76c7d516"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:40:47.260900 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.260884    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-api-access-v9xcg" (OuterVolumeSpecName: "kube-api-access-v9xcg") pod "01661e93-9bfc-40d1-a6ae-984f76c7d516" (UID: "01661e93-9bfc-40d1-a6ae-984f76c7d516"). InnerVolumeSpecName "kube-api-access-v9xcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:40:47.359612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.359531    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xcg\" (UniqueName: \"kubernetes.io/projected/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-api-access-v9xcg\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:47.359612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.359560    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:47.359612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.359570    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01661e93-9bfc-40d1-a6ae-984f76c7d516-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:47.359612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.359579    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01661e93-9bfc-40d1-a6ae-984f76c7d516-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:47.359612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:47.359587    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01661e93-9bfc-40d1-a6ae-984f76c7d516-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:40:48.007631 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.007591    2576 generic.go:356] "Generic (PLEG): container finished" podID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerID="31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622" exitCode=0
Jul 01 19:40:48.007806 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.007663    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"
Jul 01 19:40:48.007806 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.007664    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerDied","Data":"31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622"}
Jul 01 19:40:48.007806 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.007731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779" event={"ID":"01661e93-9bfc-40d1-a6ae-984f76c7d516","Type":"ContainerDied","Data":"37139ef65faae1062cbad0152b756112d5cd925091f62b282d96302794fb39ba"}
Jul 01 19:40:48.007806 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.007792    2576 scope.go:117] "RemoveContainer" containerID="31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622"
Jul 01 19:40:48.016032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.015993    2576 scope.go:117] "RemoveContainer" containerID="4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5"
Jul 01 19:40:48.023039 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.023024    2576 scope.go:117] "RemoveContainer" containerID="ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1"
Jul 01 19:40:48.030072 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.030044    2576 scope.go:117] "RemoveContainer" containerID="85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2"
Jul 01 19:40:48.030984 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.030966    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"]
Jul 01 19:40:48.034779 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.034759    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779"]
Jul 01 19:40:48.037347 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.037333    2576 scope.go:117] "RemoveContainer" containerID="31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622"
Jul 01 19:40:48.037558 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:40:48.037542    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622\": container with ID starting with 31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622 not found: ID does not exist" containerID="31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622"
Jul 01 19:40:48.037616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.037564    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622"} err="failed to get container status \"31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622\": rpc error: code = NotFound desc = could not find container \"31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622\": container with ID starting with 31b56302920d4e859981ff42b45384208c18028f87f92c98bdac30fc07f63622 not found: ID does not exist"
Jul 01 19:40:48.037616 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.037582    2576 scope.go:117] "RemoveContainer" containerID="4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5"
Jul 01 19:40:48.037795 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:40:48.037780    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5\": container with ID starting with 4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5 not found: ID does not exist" containerID="4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5"
Jul 01 19:40:48.037835 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.037801    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5"} err="failed to get container status \"4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5\": rpc error: code = NotFound desc = could not find container \"4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5\": container with ID starting with 4b70a3a2c0f85a072c07f9d951fb38dbb44d3f160ee77358bbade3c8f56fd0d5 not found: ID does not exist"
Jul 01 19:40:48.037835 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.037813    2576 scope.go:117] "RemoveContainer" containerID="ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1"
Jul 01 19:40:48.038060 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:40:48.038045    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1\": container with ID starting with ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1 not found: ID does not exist" containerID="ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1"
Jul 01 19:40:48.038116 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.038064    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1"} err="failed to get container status \"ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1\": rpc error: code = NotFound desc = could not find container \"ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1\": container with ID starting with ebcc910d7b26d43c811b5eb38f281342a3ebc1231618126ae1251e8f02ddc4d1 not found: ID does not exist"
Jul 01 19:40:48.038116 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.038076    2576 scope.go:117] "RemoveContainer" containerID="85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2"
Jul 01 19:40:48.038287 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:40:48.038273    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2\": container with ID starting with 85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2 not found: ID does not exist" containerID="85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2"
Jul 01 19:40:48.038325 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.038290    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2"} err="failed to get container status \"85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2\": rpc error: code = NotFound desc = could not find container \"85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2\": container with ID starting with 85603d22f7c612ef50b90722a69c607149b676f2ec43c348b8da04bd1e3480c2 not found: ID does not exist"
Jul 01 19:40:48.797510 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.797477    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" path="/var/lib/kubelet/pods/01661e93-9bfc-40d1-a6ae-984f76c7d516/volumes"
Jul 01 19:40:48.926691 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:48.926646    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:58.926981 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:40:58.926939    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:41:08.926235 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:41:08.926189    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:41:18.926377 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:41:18.926336    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:41:28.926175 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:41:28.926135    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:41:38.927052 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:41:38.926966    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:41:44.790621 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:41:44.790578    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:41:54.790717 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:41:54.790676    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:42:04.790862 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:04.790822    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:42:14.791178 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:14.791129    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:42:24.791159 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:24.791115    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:42:34.790518 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:34.790477    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:42:44.790610 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:44.790563    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:42:54.793778 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:54.793752    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:42:57.143833 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:57.143802    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"]
Jul 01 19:42:57.146236 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:57.144039    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" containerID="cri-o://d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b" gracePeriod=30
Jul 01 19:42:57.146236 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:57.144067    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kube-rbac-proxy" containerID="cri-o://1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c" gracePeriod=30
Jul 01 19:42:57.465647 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:57.465567    2576 generic.go:356] "Generic (PLEG): container finished" podID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerID="1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c" exitCode=2
Jul 01 19:42:57.465647 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:57.465601    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerDied","Data":"1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c"}
Jul 01 19:42:58.921844 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:42:58.921802    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 19:43:02.312813 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.312776    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"]
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313057    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313068    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313080    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313085    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313098    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="storage-initializer"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313104    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="storage-initializer"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313111    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kube-rbac-proxy"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313116    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kube-rbac-proxy"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313127    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313132    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313139    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kserve-container"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313144    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kserve-container"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313184    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kube-rbac-proxy"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313193    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kserve-container"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313198    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5923fb80-b30c-41c3-981a-553eb825e384" containerName="kube-rbac-proxy"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313204    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="agent"
Jul 01 19:43:02.313231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.313210    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01661e93-9bfc-40d1-a6ae-984f76c7d516" containerName="kserve-container"
Jul 01 19:43:02.316183 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.316167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.318729 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.318707    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-771967-predictor-serving-cert\""
Jul 01 19:43:02.318729 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.318711    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-771967-kube-rbac-proxy-sar-config\""
Jul 01 19:43:02.327289 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.327264    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"]
Jul 01 19:43:02.397413 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.397374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhs9l\" (UniqueName: \"kubernetes.io/projected/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-api-access-vhs9l\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.397413 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.397416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.397676 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.397520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.397676 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.397571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fc83dd7-b10c-4aa2-8634-31645784997e-kserve-provision-location\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.397676 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.397618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-cabundle-cert\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498016 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.497980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498016 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.498021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fc83dd7-b10c-4aa2-8634-31645784997e-kserve-provision-location\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.498043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-cabundle-cert\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.498064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhs9l\" (UniqueName: \"kubernetes.io/projected/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-api-access-vhs9l\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.498086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498264 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:43:02.498214    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-771967-predictor-serving-cert: secret "isvc-primary-771967-predictor-serving-cert" not found
Jul 01 19:43:02.498529 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:43:02.498284    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls podName:7fc83dd7-b10c-4aa2-8634-31645784997e nodeName:}" failed. No retries permitted until 2026-07-01 19:43:02.998263217 +0000 UTC m=+1250.611306598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls") pod "isvc-primary-771967-predictor-568fd899c9-8lrrm" (UID: "7fc83dd7-b10c-4aa2-8634-31645784997e") : secret "isvc-primary-771967-predictor-serving-cert" not found
Jul 01 19:43:02.498529 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.498508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fc83dd7-b10c-4aa2-8634-31645784997e-kserve-provision-location\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498767 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.498747    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-cabundle-cert\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.498869 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.498851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:02.506872 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:02.506846    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhs9l\" (UniqueName: \"kubernetes.io/projected/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-api-access-vhs9l\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:03.002412 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:03.002364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:03.004742 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:03.004722    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls\") pod \"isvc-primary-771967-predictor-568fd899c9-8lrrm\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") " pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:03.226729 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:03.226685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:03.359425 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:03.359388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"]
Jul 01 19:43:03.363400 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:43:03.363373    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc83dd7_b10c_4aa2_8634_31645784997e.slice/crio-4a2452b0d986ce98a6f299eaa6353a2b60fe5b5da70a8889b0428c9eec1c0e59 WatchSource:0}: Error finding container 4a2452b0d986ce98a6f299eaa6353a2b60fe5b5da70a8889b0428c9eec1c0e59: Status 404 returned error can't find the container with id 4a2452b0d986ce98a6f299eaa6353a2b60fe5b5da70a8889b0428c9eec1c0e59
Jul 01 19:43:03.488545 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:03.488509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerStarted","Data":"196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d"}
Jul 01 19:43:03.488737 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:03.488565    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerStarted","Data":"4a2452b0d986ce98a6f299eaa6353a2b60fe5b5da70a8889b0428c9eec1c0e59"}
Jul 01 19:43:03.921248 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:03.921198    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 19:43:04.790633 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:04.790594    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:43:07.083402 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.083366    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:43:07.234218 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234190    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-rbac-proxy-sar-config\") pod \"324ad3ad-1875-4ac1-af68-d8048aef08df\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") "
Jul 01 19:43:07.234329 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234247    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324ad3ad-1875-4ac1-af68-d8048aef08df-kserve-provision-location\") pod \"324ad3ad-1875-4ac1-af68-d8048aef08df\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") "
Jul 01 19:43:07.234329 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234285    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324ad3ad-1875-4ac1-af68-d8048aef08df-proxy-tls\") pod \"324ad3ad-1875-4ac1-af68-d8048aef08df\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") "
Jul 01 19:43:07.234418 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234335    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqzpl\" (UniqueName: \"kubernetes.io/projected/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-api-access-xqzpl\") pod \"324ad3ad-1875-4ac1-af68-d8048aef08df\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") "
Jul 01 19:43:07.234418 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234371    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-cabundle-cert\") pod \"324ad3ad-1875-4ac1-af68-d8048aef08df\" (UID: \"324ad3ad-1875-4ac1-af68-d8048aef08df\") "
Jul 01 19:43:07.234600 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234565    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324ad3ad-1875-4ac1-af68-d8048aef08df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "324ad3ad-1875-4ac1-af68-d8048aef08df" (UID: "324ad3ad-1875-4ac1-af68-d8048aef08df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:43:07.234674 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234617    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "324ad3ad-1875-4ac1-af68-d8048aef08df" (UID: "324ad3ad-1875-4ac1-af68-d8048aef08df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:43:07.234886 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.234863    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "324ad3ad-1875-4ac1-af68-d8048aef08df" (UID: "324ad3ad-1875-4ac1-af68-d8048aef08df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:43:07.236410 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.236389    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-api-access-xqzpl" (OuterVolumeSpecName: "kube-api-access-xqzpl") pod "324ad3ad-1875-4ac1-af68-d8048aef08df" (UID: "324ad3ad-1875-4ac1-af68-d8048aef08df"). InnerVolumeSpecName "kube-api-access-xqzpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:43:07.236478 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.236407    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324ad3ad-1875-4ac1-af68-d8048aef08df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "324ad3ad-1875-4ac1-af68-d8048aef08df" (UID: "324ad3ad-1875-4ac1-af68-d8048aef08df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:43:07.335050 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.335014    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqzpl\" (UniqueName: \"kubernetes.io/projected/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-api-access-xqzpl\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:43:07.335050 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.335045    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:43:07.335050 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.335055    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324ad3ad-1875-4ac1-af68-d8048aef08df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:43:07.335269 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.335065    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324ad3ad-1875-4ac1-af68-d8048aef08df-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:43:07.335269 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.335074    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324ad3ad-1875-4ac1-af68-d8048aef08df-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:43:07.504493 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.504469    2576 generic.go:356] "Generic (PLEG): container finished" podID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerID="d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b" exitCode=0
Jul 01 19:43:07.504623 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.504529    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"
Jul 01 19:43:07.504623 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.504555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerDied","Data":"d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b"}
Jul 01 19:43:07.504715 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.504628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2" event={"ID":"324ad3ad-1875-4ac1-af68-d8048aef08df","Type":"ContainerDied","Data":"7e3ee86f4d276b5fbef6fbf88c7ce576dae126865df6ffb4471eec21e9dac139"}
Jul 01 19:43:07.504715 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.504667    2576 scope.go:117] "RemoveContainer" containerID="1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c"
Jul 01 19:43:07.506210 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.506184    2576 generic.go:356] "Generic (PLEG): container finished" podID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerID="196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d" exitCode=0
Jul 01 19:43:07.506321 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.506234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerDied","Data":"196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d"}
Jul 01 19:43:07.513031 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.512979    2576 scope.go:117] "RemoveContainer" containerID="d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b"
Jul 01 19:43:07.520044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.520027    2576 scope.go:117] "RemoveContainer" containerID="ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8"
Jul 01 19:43:07.528620 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.528603    2576 scope.go:117] "RemoveContainer" containerID="1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c"
Jul 01 19:43:07.528868 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:43:07.528849    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c\": container with ID starting with 1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c not found: ID does not exist" containerID="1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c"
Jul 01 19:43:07.528950 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.528876    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c"} err="failed to get container status \"1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c\": rpc error: code = NotFound desc = could not find container \"1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c\": container with ID starting with 1c87a2b990a4d12c02dedf8e546220ad30931ef0e33eca3d93b85c832659891c not found: ID does not exist"
Jul 01 19:43:07.528950 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.528893    2576 scope.go:117] "RemoveContainer" containerID="d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b"
Jul 01 19:43:07.529177 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:43:07.529156    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b\": container with ID starting with d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b not found: ID does not exist" containerID="d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b"
Jul 01 19:43:07.529239 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.529187    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b"} err="failed to get container status \"d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b\": rpc error: code = NotFound desc = could not find container \"d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b\": container with ID starting with d95df88bf6014507b3028e20a6fe718e77cd4aebf7220bbcd93e83bde78ddb5b not found: ID does not exist"
Jul 01 19:43:07.529239 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.529210    2576 scope.go:117] "RemoveContainer" containerID="ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8"
Jul 01 19:43:07.529473 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:43:07.529456    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8\": container with ID starting with ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8 not found: ID does not exist" containerID="ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8"
Jul 01 19:43:07.529520 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.529479    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8"} err="failed to get container status \"ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8\": rpc error: code = NotFound desc = could not find container \"ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8\": container with ID starting with ec6dade99367bc348da8ef37b3b2761fbeb0f364cdf97bc49d165226e9f85cd8 not found: ID does not exist"
Jul 01 19:43:07.539945 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.539906    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"]
Jul 01 19:43:07.544339 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:07.544317    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2"]
Jul 01 19:43:08.512681 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:08.512648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerStarted","Data":"c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447"}
Jul 01 19:43:08.513091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:08.512704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerStarted","Data":"3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e"}
Jul 01 19:43:08.513091 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:08.512971    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:08.513165 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:08.513096    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:08.514375 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:08.514352    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:43:08.533975 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:08.533909    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podStartSLOduration=2.737021182 podStartE2EDuration="6.533893652s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79687247s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:08.53297133 +0000 UTC m=+1256.146014718" watchObservedRunningTime="2026-07-01 19:43:08.533893652 +0000 UTC m=+1256.146937039"
Jul 01 19:43:08.794433 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:08.794355    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" path="/var/lib/kubelet/pods/324ad3ad-1875-4ac1-af68-d8048aef08df/volumes"
Jul 01 19:43:09.517104 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:09.517058    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:43:14.521875 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:14.521847    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:43:14.522352 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:14.522324    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:43:24.523323 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:24.523285    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:43:34.522594 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:34.522555    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:43:44.522411 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:44.522366    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:43:54.522889 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:43:54.522850    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:44:04.522730 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:04.522687    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:44:14.522986 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:14.522945    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:44:17.790570 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:17.790526    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 19:44:27.791785 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:27.791749    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:44:32.539294 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539253    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"]
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539551    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container"
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539562    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container"
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539569    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kube-rbac-proxy"
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539574    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kube-rbac-proxy"
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539583    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="storage-initializer"
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539589    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="storage-initializer"
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539635    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kube-rbac-proxy"
Jul 01 19:44:32.539716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.539644    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="324ad3ad-1875-4ac1-af68-d8048aef08df" containerName="kserve-container"
Jul 01 19:44:32.542689 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.542672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.545336 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.545317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-771967-predictor-serving-cert\""
Jul 01 19:44:32.545491 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.545469    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-771967\""
Jul 01 19:44:32.545491 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.545484    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-771967-dockercfg-jb8jc\""
Jul 01 19:44:32.545586 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.545481    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-771967-kube-rbac-proxy-sar-config\""
Jul 01 19:44:32.558134 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.558112    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"]
Jul 01 19:44:32.589888 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.589859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac18698-274f-479b-b3f0-4e896c1e1211-kserve-provision-location\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.590054 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.589950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xkz\" (UniqueName: \"kubernetes.io/projected/eac18698-274f-479b-b3f0-4e896c1e1211-kube-api-access-v8xkz\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.590054 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.589979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac18698-274f-479b-b3f0-4e896c1e1211-proxy-tls\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.590054 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.590008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.590054 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.590030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-cabundle-cert\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.690830 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.690796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac18698-274f-479b-b3f0-4e896c1e1211-kserve-provision-location\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.691035 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.690856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xkz\" (UniqueName: \"kubernetes.io/projected/eac18698-274f-479b-b3f0-4e896c1e1211-kube-api-access-v8xkz\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.691035 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.690895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac18698-274f-479b-b3f0-4e896c1e1211-proxy-tls\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.691154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.691036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.691154 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.691075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-cabundle-cert\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.691297 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.691270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac18698-274f-479b-b3f0-4e896c1e1211-kserve-provision-location\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.691712 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.691691    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-cabundle-cert\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.691790 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.691714    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.693270 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.693253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac18698-274f-479b-b3f0-4e896c1e1211-proxy-tls\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.705228 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.705203    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xkz\" (UniqueName: \"kubernetes.io/projected/eac18698-274f-479b-b3f0-4e896c1e1211-kube-api-access-v8xkz\") pod \"isvc-secondary-771967-predictor-85df677c64-w8fhw\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") " pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.853022 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.852939    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:32.982684 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:32.982597    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"]
Jul 01 19:44:32.984812 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:44:32.984785    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac18698_274f_479b_b3f0_4e896c1e1211.slice/crio-658092f47d66d2314c3ff6ee2ecefde01535323ec3ee9d5023ea41b2067247b3 WatchSource:0}: Error finding container 658092f47d66d2314c3ff6ee2ecefde01535323ec3ee9d5023ea41b2067247b3: Status 404 returned error can't find the container with id 658092f47d66d2314c3ff6ee2ecefde01535323ec3ee9d5023ea41b2067247b3
Jul 01 19:44:33.822224 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:33.822187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw" event={"ID":"eac18698-274f-479b-b3f0-4e896c1e1211","Type":"ContainerStarted","Data":"e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338"}
Jul 01 19:44:33.822596 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:33.822270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw" event={"ID":"eac18698-274f-479b-b3f0-4e896c1e1211","Type":"ContainerStarted","Data":"658092f47d66d2314c3ff6ee2ecefde01535323ec3ee9d5023ea41b2067247b3"}
Jul 01 19:44:37.837653 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:37.837624    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-771967-predictor-85df677c64-w8fhw_eac18698-274f-479b-b3f0-4e896c1e1211/storage-initializer/0.log"
Jul 01 19:44:37.838074 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:37.837660    2576 generic.go:356] "Generic (PLEG): container finished" podID="eac18698-274f-479b-b3f0-4e896c1e1211" containerID="e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338" exitCode=1
Jul 01 19:44:37.838074 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:37.837692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw" event={"ID":"eac18698-274f-479b-b3f0-4e896c1e1211","Type":"ContainerDied","Data":"e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338"}
Jul 01 19:44:38.843227 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:38.843199    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-771967-predictor-85df677c64-w8fhw_eac18698-274f-479b-b3f0-4e896c1e1211/storage-initializer/0.log"
Jul 01 19:44:38.843227 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:38.843243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw" event={"ID":"eac18698-274f-479b-b3f0-4e896c1e1211","Type":"ContainerStarted","Data":"cafb61afa59b9f029060964436bbad8fe26affe69fcc5e2ab78de25deb76d917"}
Jul 01 19:44:43.861852 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:43.861816    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-771967-predictor-85df677c64-w8fhw_eac18698-274f-479b-b3f0-4e896c1e1211/storage-initializer/1.log"
Jul 01 19:44:43.862296 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:43.862208    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-771967-predictor-85df677c64-w8fhw_eac18698-274f-479b-b3f0-4e896c1e1211/storage-initializer/0.log"
Jul 01 19:44:43.862296 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:43.862241    2576 generic.go:356] "Generic (PLEG): container finished" podID="eac18698-274f-479b-b3f0-4e896c1e1211" containerID="cafb61afa59b9f029060964436bbad8fe26affe69fcc5e2ab78de25deb76d917" exitCode=1
Jul 01 19:44:43.862367 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:43.862320    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw" event={"ID":"eac18698-274f-479b-b3f0-4e896c1e1211","Type":"ContainerDied","Data":"cafb61afa59b9f029060964436bbad8fe26affe69fcc5e2ab78de25deb76d917"}
Jul 01 19:44:43.862404 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:43.862382    2576 scope.go:117] "RemoveContainer" containerID="e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338"
Jul 01 19:44:43.862786 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:43.862767    2576 scope.go:117] "RemoveContainer" containerID="e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338"
Jul 01 19:44:43.872466 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:44:43.872442    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-771967-predictor-85df677c64-w8fhw_kserve-ci-e2e-test(eac18698-274f-479b-b3f0-4e896c1e1211)\"" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw" podUID="eac18698-274f-479b-b3f0-4e896c1e1211"
Jul 01 19:44:43.872602 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:44:43.872574    2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-771967-predictor-85df677c64-w8fhw_kserve-ci-e2e-test_eac18698-274f-479b-b3f0-4e896c1e1211_0 in pod sandbox 658092f47d66d2314c3ff6ee2ecefde01535323ec3ee9d5023ea41b2067247b3 from index: no such id: 'e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338'" containerID="e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338"
Jul 01 19:44:43.872656 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:43.872613    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-771967-predictor-85df677c64-w8fhw_kserve-ci-e2e-test_eac18698-274f-479b-b3f0-4e896c1e1211_0 in pod sandbox 658092f47d66d2314c3ff6ee2ecefde01535323ec3ee9d5023ea41b2067247b3 from index: no such id: 'e6d0d71126001cd4481f85b451ffb9945ee53aaa435e74f20bc1aa1b63341338'"
Jul 01 19:44:44.870221 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:44.870197    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-771967-predictor-85df677c64-w8fhw_eac18698-274f-479b-b3f0-4e896c1e1211/storage-initializer/1.log"
Jul 01 19:44:48.543937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.543880    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"]
Jul 01 19:44:48.544352 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.544116    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container" containerID="cri-o://3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e" gracePeriod=30
Jul 01 19:44:48.544352 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.544178    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kube-rbac-proxy" containerID="cri-o://c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447" gracePeriod=30
Jul 01 19:44:48.612906 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.612876    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"]
Jul 01 19:44:48.744851 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.744827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-771967-predictor-85df677c64-w8fhw_eac18698-274f-479b-b3f0-4e896c1e1211/storage-initializer/1.log"
Jul 01 19:44:48.747177 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.747162    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:48.808172 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac18698-274f-479b-b3f0-4e896c1e1211-proxy-tls\") pod \"eac18698-274f-479b-b3f0-4e896c1e1211\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") "
Jul 01 19:44:48.808172 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-kube-rbac-proxy-sar-config\") pod \"eac18698-274f-479b-b3f0-4e896c1e1211\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") "
Jul 01 19:44:48.808374 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8xkz\" (UniqueName: \"kubernetes.io/projected/eac18698-274f-479b-b3f0-4e896c1e1211-kube-api-access-v8xkz\") pod \"eac18698-274f-479b-b3f0-4e896c1e1211\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") "
Jul 01 19:44:48.808374 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808229    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac18698-274f-479b-b3f0-4e896c1e1211-kserve-provision-location\") pod \"eac18698-274f-479b-b3f0-4e896c1e1211\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") "
Jul 01 19:44:48.808374 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-cabundle-cert\") pod \"eac18698-274f-479b-b3f0-4e896c1e1211\" (UID: \"eac18698-274f-479b-b3f0-4e896c1e1211\") "
Jul 01 19:44:48.808533 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808457    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac18698-274f-479b-b3f0-4e896c1e1211-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eac18698-274f-479b-b3f0-4e896c1e1211" (UID: "eac18698-274f-479b-b3f0-4e896c1e1211"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:44:48.808585 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808557    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eac18698-274f-479b-b3f0-4e896c1e1211" (UID: "eac18698-274f-479b-b3f0-4e896c1e1211"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:48.808628 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.808605    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eac18698-274f-479b-b3f0-4e896c1e1211" (UID: "eac18698-274f-479b-b3f0-4e896c1e1211"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:48.810243 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.810219    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac18698-274f-479b-b3f0-4e896c1e1211-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eac18698-274f-479b-b3f0-4e896c1e1211" (UID: "eac18698-274f-479b-b3f0-4e896c1e1211"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:44:48.810388 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.810373    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac18698-274f-479b-b3f0-4e896c1e1211-kube-api-access-v8xkz" (OuterVolumeSpecName: "kube-api-access-v8xkz") pod "eac18698-274f-479b-b3f0-4e896c1e1211" (UID: "eac18698-274f-479b-b3f0-4e896c1e1211"). InnerVolumeSpecName "kube-api-access-v8xkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:44:48.886970 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.886940    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-771967-predictor-85df677c64-w8fhw_eac18698-274f-479b-b3f0-4e896c1e1211/storage-initializer/1.log"
Jul 01 19:44:48.887122 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.887044    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"
Jul 01 19:44:48.887122 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.887081    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw" event={"ID":"eac18698-274f-479b-b3f0-4e896c1e1211","Type":"ContainerDied","Data":"658092f47d66d2314c3ff6ee2ecefde01535323ec3ee9d5023ea41b2067247b3"}
Jul 01 19:44:48.887220 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.887159    2576 scope.go:117] "RemoveContainer" containerID="cafb61afa59b9f029060964436bbad8fe26affe69fcc5e2ab78de25deb76d917"
Jul 01 19:44:48.889396 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.889372    2576 generic.go:356] "Generic (PLEG): container finished" podID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerID="c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447" exitCode=2
Jul 01 19:44:48.889511 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.889408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerDied","Data":"c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447"}
Jul 01 19:44:48.909718 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.909693    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eac18698-274f-479b-b3f0-4e896c1e1211-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:48.909718 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.909712    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:48.909845 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.909722    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xkz\" (UniqueName: \"kubernetes.io/projected/eac18698-274f-479b-b3f0-4e896c1e1211-kube-api-access-v8xkz\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:48.909845 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.909731    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eac18698-274f-479b-b3f0-4e896c1e1211-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:48.909845 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.909739    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eac18698-274f-479b-b3f0-4e896c1e1211-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:48.928961 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.928934    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"]
Jul 01 19:44:48.933706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:48.933676    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw"]
Jul 01 19:44:49.517714 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:49.517676    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 01 19:44:50.794316 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:50.794286    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac18698-274f-479b-b3f0-4e896c1e1211" path="/var/lib/kubelet/pods/eac18698-274f-479b-b3f0-4e896c1e1211/volumes"
Jul 01 19:44:53.193598 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.193569    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:44:53.240839 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.240810    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fc83dd7-b10c-4aa2-8634-31645784997e-kserve-provision-location\") pod \"7fc83dd7-b10c-4aa2-8634-31645784997e\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") "
Jul 01 19:44:53.241032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.240846    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls\") pod \"7fc83dd7-b10c-4aa2-8634-31645784997e\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") "
Jul 01 19:44:53.241032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.240874    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-rbac-proxy-sar-config\") pod \"7fc83dd7-b10c-4aa2-8634-31645784997e\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") "
Jul 01 19:44:53.241032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.240933    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-cabundle-cert\") pod \"7fc83dd7-b10c-4aa2-8634-31645784997e\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") "
Jul 01 19:44:53.241032 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.240954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhs9l\" (UniqueName: \"kubernetes.io/projected/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-api-access-vhs9l\") pod \"7fc83dd7-b10c-4aa2-8634-31645784997e\" (UID: \"7fc83dd7-b10c-4aa2-8634-31645784997e\") "
Jul 01 19:44:53.241240 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.241219    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc83dd7-b10c-4aa2-8634-31645784997e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fc83dd7-b10c-4aa2-8634-31645784997e" (UID: "7fc83dd7-b10c-4aa2-8634-31645784997e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:44:53.241332 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.241283    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fc83dd7-b10c-4aa2-8634-31645784997e" (UID: "7fc83dd7-b10c-4aa2-8634-31645784997e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:53.241332 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.241287    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7fc83dd7-b10c-4aa2-8634-31645784997e" (UID: "7fc83dd7-b10c-4aa2-8634-31645784997e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:53.242961 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.242937    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fc83dd7-b10c-4aa2-8634-31645784997e" (UID: "7fc83dd7-b10c-4aa2-8634-31645784997e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:44:53.243044 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.242998    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-api-access-vhs9l" (OuterVolumeSpecName: "kube-api-access-vhs9l") pod "7fc83dd7-b10c-4aa2-8634-31645784997e" (UID: "7fc83dd7-b10c-4aa2-8634-31645784997e"). InnerVolumeSpecName "kube-api-access-vhs9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:44:53.342372 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.342287    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:53.342372 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.342318    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fc83dd7-b10c-4aa2-8634-31645784997e-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:53.342372 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.342330    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhs9l\" (UniqueName: \"kubernetes.io/projected/7fc83dd7-b10c-4aa2-8634-31645784997e-kube-api-access-vhs9l\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:53.342372 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.342339    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fc83dd7-b10c-4aa2-8634-31645784997e-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:53.342372 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.342348    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fc83dd7-b10c-4aa2-8634-31645784997e-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:44:53.757648 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757615    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"]
Jul 01 19:44:53.757874 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757863    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac18698-274f-479b-b3f0-4e896c1e1211" containerName="storage-initializer"
Jul 01 19:44:53.757937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757876    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac18698-274f-479b-b3f0-4e896c1e1211" containerName="storage-initializer"
Jul 01 19:44:53.757937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757890    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kube-rbac-proxy"
Jul 01 19:44:53.757937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757896    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kube-rbac-proxy"
Jul 01 19:44:53.757937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757907    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="storage-initializer"
Jul 01 19:44:53.757937 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757926    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="storage-initializer"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757939    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757947    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.757993    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kserve-container"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.758000    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac18698-274f-479b-b3f0-4e896c1e1211" containerName="storage-initializer"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.758006    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerName="kube-rbac-proxy"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.758012    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac18698-274f-479b-b3f0-4e896c1e1211" containerName="storage-initializer"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.758063    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac18698-274f-479b-b3f0-4e896c1e1211" containerName="storage-initializer"
Jul 01 19:44:53.758103 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.758069    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac18698-274f-479b-b3f0-4e896c1e1211" containerName="storage-initializer"
Jul 01 19:44:53.761149 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.761133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.764683 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.764666    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e38b18\""
Jul 01 19:44:53.764895 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.764877    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e38b18-kube-rbac-proxy-sar-config\""
Jul 01 19:44:53.765800 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.765783    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e38b18-predictor-serving-cert\""
Jul 01 19:44:53.765868 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.765819    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e38b18-dockercfg-z5cdk\""
Jul 01 19:44:53.779733 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.779713    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"]
Jul 01 19:44:53.845692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.845653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxp7v\" (UniqueName: \"kubernetes.io/projected/2a265987-19b9-469d-a531-d15188ba5a5f-kube-api-access-rxp7v\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.845692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.845693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a265987-19b9-469d-a531-d15188ba5a5f-kserve-provision-location\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.845872 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.845714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-cabundle-cert\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.845872 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.845774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.845872 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.845807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.909598 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.909562    2576 generic.go:356] "Generic (PLEG): container finished" podID="7fc83dd7-b10c-4aa2-8634-31645784997e" containerID="3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e" exitCode=0
Jul 01 19:44:53.909598 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.909597    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerDied","Data":"3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e"}
Jul 01 19:44:53.909800 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.909622    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"
Jul 01 19:44:53.909800 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.909644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm" event={"ID":"7fc83dd7-b10c-4aa2-8634-31645784997e","Type":"ContainerDied","Data":"4a2452b0d986ce98a6f299eaa6353a2b60fe5b5da70a8889b0428c9eec1c0e59"}
Jul 01 19:44:53.909800 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.909671    2576 scope.go:117] "RemoveContainer" containerID="c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447"
Jul 01 19:44:53.917853 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.917829    2576 scope.go:117] "RemoveContainer" containerID="3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e"
Jul 01 19:44:53.924822 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.924805    2576 scope.go:117] "RemoveContainer" containerID="196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d"
Jul 01 19:44:53.931661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.931640    2576 scope.go:117] "RemoveContainer" containerID="c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447"
Jul 01 19:44:53.931951 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:44:53.931911    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447\": container with ID starting with c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447 not found: ID does not exist" containerID="c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447"
Jul 01 19:44:53.932024 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.931959    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447"} err="failed to get container status \"c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447\": rpc error: code = NotFound desc = could not find container \"c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447\": container with ID starting with c7fe3a6610cc9e16560043ecacbd8c14604c35ecdc270c9dff2459eac9a93447 not found: ID does not exist"
Jul 01 19:44:53.932024 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.931976    2576 scope.go:117] "RemoveContainer" containerID="3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e"
Jul 01 19:44:53.932250 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:44:53.932227    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e\": container with ID starting with 3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e not found: ID does not exist" containerID="3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e"
Jul 01 19:44:53.932290 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.932262    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e"} err="failed to get container status \"3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e\": rpc error: code = NotFound desc = could not find container \"3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e\": container with ID starting with 3344c229129b78e9303ec268cec0e9cf3d45bec799a738203b48e4ab6d8b088e not found: ID does not exist"
Jul 01 19:44:53.932290 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.932285    2576 scope.go:117] "RemoveContainer" containerID="196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d"
Jul 01 19:44:53.932555 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:44:53.932536    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d\": container with ID starting with 196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d not found: ID does not exist" containerID="196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d"
Jul 01 19:44:53.932595 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.932564    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d"} err="failed to get container status \"196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d\": rpc error: code = NotFound desc = could not find container \"196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d\": container with ID starting with 196cd9c6c704f726c0b94923e81c0f2fe1ed62e4aa89c9ce8c67f0043125a59d not found: ID does not exist"
Jul 01 19:44:53.932958 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.932937    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"]
Jul 01 19:44:53.936740 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.936722    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm"]
Jul 01 19:44:53.946848 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.946826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.946959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.946861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.946959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.946892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxp7v\" (UniqueName: \"kubernetes.io/projected/2a265987-19b9-469d-a531-d15188ba5a5f-kube-api-access-rxp7v\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.946959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.946945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a265987-19b9-469d-a531-d15188ba5a5f-kserve-provision-location\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.947125 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.946975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-cabundle-cert\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.947125 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:44:53.947008    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-serving-cert: secret "isvc-init-fail-e38b18-predictor-serving-cert" not found
Jul 01 19:44:53.947125 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:44:53.947087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls podName:2a265987-19b9-469d-a531-d15188ba5a5f nodeName:}" failed. No retries permitted until 2026-07-01 19:44:54.447056994 +0000 UTC m=+1362.060100373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls") pod "isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" (UID: "2a265987-19b9-469d-a531-d15188ba5a5f") : secret "isvc-init-fail-e38b18-predictor-serving-cert" not found
Jul 01 19:44:53.947347 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.947323    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a265987-19b9-469d-a531-d15188ba5a5f-kserve-provision-location\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.947576 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.947557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.947634 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.947585    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-cabundle-cert\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:53.959011 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:53.958983    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxp7v\" (UniqueName: \"kubernetes.io/projected/2a265987-19b9-469d-a531-d15188ba5a5f-kube-api-access-rxp7v\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:54.451301 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:54.451264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:54.453612 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:54.453591    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls\") pod \"isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") " pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:54.671586 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:54.671540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:44:54.794741 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:54.794709    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc83dd7-b10c-4aa2-8634-31645784997e" path="/var/lib/kubelet/pods/7fc83dd7-b10c-4aa2-8634-31645784997e/volumes"
Jul 01 19:44:54.802162 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:54.802136    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"]
Jul 01 19:44:54.804814 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:44:54.804792    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a265987_19b9_469d_a531_d15188ba5a5f.slice/crio-07eed20719c714a28e09156340fc0a1138c0bcd450292373ae03ac173eb592ea WatchSource:0}: Error finding container 07eed20719c714a28e09156340fc0a1138c0bcd450292373ae03ac173eb592ea: Status 404 returned error can't find the container with id 07eed20719c714a28e09156340fc0a1138c0bcd450292373ae03ac173eb592ea
Jul 01 19:44:54.915697 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:54.915659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" event={"ID":"2a265987-19b9-469d-a531-d15188ba5a5f","Type":"ContainerStarted","Data":"4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90"}
Jul 01 19:44:54.915875 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:44:54.915764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" event={"ID":"2a265987-19b9-469d-a531-d15188ba5a5f","Type":"ContainerStarted","Data":"07eed20719c714a28e09156340fc0a1138c0bcd450292373ae03ac173eb592ea"}
Jul 01 19:45:00.940802 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:00.940771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg_2a265987-19b9-469d-a531-d15188ba5a5f/storage-initializer/0.log"
Jul 01 19:45:00.941200 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:00.940813    2576 generic.go:356] "Generic (PLEG): container finished" podID="2a265987-19b9-469d-a531-d15188ba5a5f" containerID="4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90" exitCode=1
Jul 01 19:45:00.941200 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:00.940887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" event={"ID":"2a265987-19b9-469d-a531-d15188ba5a5f","Type":"ContainerDied","Data":"4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90"}
Jul 01 19:45:01.946867 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:01.946841    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg_2a265987-19b9-469d-a531-d15188ba5a5f/storage-initializer/0.log"
Jul 01 19:45:01.947276 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:01.946932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" event={"ID":"2a265987-19b9-469d-a531-d15188ba5a5f","Type":"ContainerStarted","Data":"037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0"}
Jul 01 19:45:03.784116 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:03.784086    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"]
Jul 01 19:45:03.784509 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:03.784320    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" containerName="storage-initializer" containerID="cri-o://037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0" gracePeriod=30
Jul 01 19:45:04.425058 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.425036    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg_2a265987-19b9-469d-a531-d15188ba5a5f/storage-initializer/1.log"
Jul 01 19:45:04.425479 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.425463    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg_2a265987-19b9-469d-a531-d15188ba5a5f/storage-initializer/0.log"
Jul 01 19:45:04.427781 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.427766    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:45:04.528229 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528200    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-cabundle-cert\") pod \"2a265987-19b9-469d-a531-d15188ba5a5f\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") "
Jul 01 19:45:04.528385 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528239    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxp7v\" (UniqueName: \"kubernetes.io/projected/2a265987-19b9-469d-a531-d15188ba5a5f-kube-api-access-rxp7v\") pod \"2a265987-19b9-469d-a531-d15188ba5a5f\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") "
Jul 01 19:45:04.528385 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528267    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls\") pod \"2a265987-19b9-469d-a531-d15188ba5a5f\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") "
Jul 01 19:45:04.528385 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528324    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-kube-rbac-proxy-sar-config\") pod \"2a265987-19b9-469d-a531-d15188ba5a5f\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") "
Jul 01 19:45:04.528385 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528369    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a265987-19b9-469d-a531-d15188ba5a5f-kserve-provision-location\") pod \"2a265987-19b9-469d-a531-d15188ba5a5f\" (UID: \"2a265987-19b9-469d-a531-d15188ba5a5f\") "
Jul 01 19:45:04.528710 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528683    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a265987-19b9-469d-a531-d15188ba5a5f" (UID: "2a265987-19b9-469d-a531-d15188ba5a5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:45:04.528801 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528698    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a265987-19b9-469d-a531-d15188ba5a5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a265987-19b9-469d-a531-d15188ba5a5f" (UID: "2a265987-19b9-469d-a531-d15188ba5a5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:45:04.528801 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.528744    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a265987-19b9-469d-a531-d15188ba5a5f" (UID: "2a265987-19b9-469d-a531-d15188ba5a5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:45:04.530429 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.530401    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a265987-19b9-469d-a531-d15188ba5a5f" (UID: "2a265987-19b9-469d-a531-d15188ba5a5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:45:04.530547 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.530460    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a265987-19b9-469d-a531-d15188ba5a5f-kube-api-access-rxp7v" (OuterVolumeSpecName: "kube-api-access-rxp7v") pod "2a265987-19b9-469d-a531-d15188ba5a5f" (UID: "2a265987-19b9-469d-a531-d15188ba5a5f"). InnerVolumeSpecName "kube-api-access-rxp7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:45:04.629351 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.629313    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:45:04.629351 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.629345    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxp7v\" (UniqueName: \"kubernetes.io/projected/2a265987-19b9-469d-a531-d15188ba5a5f-kube-api-access-rxp7v\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:45:04.629558 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.629401    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a265987-19b9-469d-a531-d15188ba5a5f-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:45:04.629558 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.629412    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a265987-19b9-469d-a531-d15188ba5a5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:45:04.629558 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.629421    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a265987-19b9-469d-a531-d15188ba5a5f-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:45:04.958470 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.958444    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg_2a265987-19b9-469d-a531-d15188ba5a5f/storage-initializer/1.log"
Jul 01 19:45:04.958866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.958797    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg_2a265987-19b9-469d-a531-d15188ba5a5f/storage-initializer/0.log"
Jul 01 19:45:04.958866 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.958835    2576 generic.go:356] "Generic (PLEG): container finished" podID="2a265987-19b9-469d-a531-d15188ba5a5f" containerID="037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0" exitCode=1
Jul 01 19:45:04.958964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.958864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" event={"ID":"2a265987-19b9-469d-a531-d15188ba5a5f","Type":"ContainerDied","Data":"037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0"}
Jul 01 19:45:04.958964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.958894    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"
Jul 01 19:45:04.958964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.958933    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg" event={"ID":"2a265987-19b9-469d-a531-d15188ba5a5f","Type":"ContainerDied","Data":"07eed20719c714a28e09156340fc0a1138c0bcd450292373ae03ac173eb592ea"}
Jul 01 19:45:04.958964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.958960    2576 scope.go:117] "RemoveContainer" containerID="037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0"
Jul 01 19:45:04.966567 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.966550    2576 scope.go:117] "RemoveContainer" containerID="4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90"
Jul 01 19:45:04.973275 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.973261    2576 scope.go:117] "RemoveContainer" containerID="037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0"
Jul 01 19:45:04.973527 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:45:04.973509    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0\": container with ID starting with 037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0 not found: ID does not exist" containerID="037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0"
Jul 01 19:45:04.973580 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.973533    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0"} err="failed to get container status \"037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0\": rpc error: code = NotFound desc = could not find container \"037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0\": container with ID starting with 037061690b5621e5a1064003efff0e398c3c5b598e42310b3021729150c568b0 not found: ID does not exist"
Jul 01 19:45:04.973580 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.973554    2576 scope.go:117] "RemoveContainer" containerID="4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90"
Jul 01 19:45:04.973786 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:45:04.973768    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90\": container with ID starting with 4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90 not found: ID does not exist" containerID="4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90"
Jul 01 19:45:04.973841 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:04.973796    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90"} err="failed to get container status \"4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90\": rpc error: code = NotFound desc = could not find container \"4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90\": container with ID starting with 4f7d24fe68ce5d2a9aad1c6a765dfeb105ee7ee28d833c373374976766a8dc90 not found: ID does not exist"
Jul 01 19:45:05.005557 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:05.005509    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"]
Jul 01 19:45:05.011231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:05.011201    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg"]
Jul 01 19:45:06.794114 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:06.794085    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" path="/var/lib/kubelet/pods/2a265987-19b9-469d-a531-d15188ba5a5f/volumes"
Jul 01 19:45:08.967762 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.967726    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"]
Jul 01 19:45:08.968629 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.968606    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" containerName="storage-initializer"
Jul 01 19:45:08.968714 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.968631    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" containerName="storage-initializer"
Jul 01 19:45:08.968714 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.968667    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" containerName="storage-initializer"
Jul 01 19:45:08.968714 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.968680    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" containerName="storage-initializer"
Jul 01 19:45:08.968868 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.968854    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" containerName="storage-initializer"
Jul 01 19:45:08.968938 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.968875    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a265987-19b9-469d-a531-d15188ba5a5f" containerName="storage-initializer"
Jul 01 19:45:08.975871 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.975840    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:08.979851 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.979826    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b55a9-predictor-serving-cert\""
Jul 01 19:45:08.979851 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.979829    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:45:08.980042 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.979912    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bqdbq\""
Jul 01 19:45:08.980219 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.980201    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:45:08.980696 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.980681    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b55a9-kube-rbac-proxy-sar-config\""
Jul 01 19:45:08.980788 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.980775    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:45:08.980990 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.980968    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:45:08.985521 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:08.985499    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"]
Jul 01 19:45:09.065089 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.065047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15054085-b6ab-4c89-9db8-dd3ceebe26da-proxy-tls\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.065263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.065101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9ffs\" (UniqueName: \"kubernetes.io/projected/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-api-access-r9ffs\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.065263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.065152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.065263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.065175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15054085-b6ab-4c89-9db8-dd3ceebe26da-kserve-provision-location\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.065263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.065201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-cabundle-cert\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.166129 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15054085-b6ab-4c89-9db8-dd3ceebe26da-kserve-provision-location\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.166129 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-cabundle-cert\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.166377 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15054085-b6ab-4c89-9db8-dd3ceebe26da-proxy-tls\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.166377 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9ffs\" (UniqueName: \"kubernetes.io/projected/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-api-access-r9ffs\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.166377 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.166588 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15054085-b6ab-4c89-9db8-dd3ceebe26da-kserve-provision-location\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.166959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166936    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.167042 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.166945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-cabundle-cert\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.168617 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.168594    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15054085-b6ab-4c89-9db8-dd3ceebe26da-proxy-tls\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.175268 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.175247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9ffs\" (UniqueName: \"kubernetes.io/projected/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-api-access-r9ffs\") pod \"raw-sklearn-b55a9-predictor-5b69cff558-tnkqz\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") " pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.286195 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.286167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:09.418789 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.418763    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"]
Jul 01 19:45:09.420978 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:45:09.420947    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15054085_b6ab_4c89_9db8_dd3ceebe26da.slice/crio-85fb7f3214c0d0c99130e6f23145373fa2208491c4f419caf4db8db0c4e28590 WatchSource:0}: Error finding container 85fb7f3214c0d0c99130e6f23145373fa2208491c4f419caf4db8db0c4e28590: Status 404 returned error can't find the container with id 85fb7f3214c0d0c99130e6f23145373fa2208491c4f419caf4db8db0c4e28590
Jul 01 19:45:09.980567 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.980529    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerStarted","Data":"27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca"}
Jul 01 19:45:09.981004 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:09.980590    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerStarted","Data":"85fb7f3214c0d0c99130e6f23145373fa2208491c4f419caf4db8db0c4e28590"}
Jul 01 19:45:13.995148 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:13.995117    2576 generic.go:356] "Generic (PLEG): container finished" podID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerID="27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca" exitCode=0
Jul 01 19:45:13.995148 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:13.995151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerDied","Data":"27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca"}
Jul 01 19:45:15.000716 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:15.000679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerStarted","Data":"1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af"}
Jul 01 19:45:15.001137 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:15.000741    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerStarted","Data":"7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655"}
Jul 01 19:45:15.001137 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:15.001041    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:15.027960 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:15.027890    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podStartSLOduration=3.306086953 podStartE2EDuration="7.027875428s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721788475s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:45:15.026172017 +0000 UTC m=+1382.639215409" watchObservedRunningTime="2026-07-01 19:45:15.027875428 +0000 UTC m=+1382.640918815"
Jul 01 19:45:16.004793 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:16.004762    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:16.006119 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:16.006091    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:45:17.007955 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:17.007894    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:45:22.012948 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:22.012906    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:45:22.013390 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:22.013365    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:45:32.013416 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:32.013373    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:45:42.014204 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:42.014165    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:45:52.014316 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:45:52.014274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:46:02.013403 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:02.013362    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:46:12.013782 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:12.013746    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:46:22.014141 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:22.014103    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:46:32.014118 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:32.014086    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:46:39.104526 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:39.104495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"]
Jul 01 19:46:39.105144 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:39.104731    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" containerID="cri-o://7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655" gracePeriod=30
Jul 01 19:46:39.105144 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:39.104779    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kube-rbac-proxy" containerID="cri-o://1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af" gracePeriod=30
Jul 01 19:46:39.301814 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:39.301784    2576 generic.go:356] "Generic (PLEG): container finished" podID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerID="1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af" exitCode=2
Jul 01 19:46:39.302011 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:39.301865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerDied","Data":"1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af"}
Jul 01 19:46:42.008457 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:42.008416    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 01 19:46:42.013872 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:42.013841    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:46:43.751242 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.751204    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:46:43.910433 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910346    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15054085-b6ab-4c89-9db8-dd3ceebe26da-kserve-provision-location\") pod \"15054085-b6ab-4c89-9db8-dd3ceebe26da\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") "
Jul 01 19:46:43.910433 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910413    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-rbac-proxy-sar-config\") pod \"15054085-b6ab-4c89-9db8-dd3ceebe26da\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") "
Jul 01 19:46:43.910670 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910438    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15054085-b6ab-4c89-9db8-dd3ceebe26da-proxy-tls\") pod \"15054085-b6ab-4c89-9db8-dd3ceebe26da\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") "
Jul 01 19:46:43.910670 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910454    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9ffs\" (UniqueName: \"kubernetes.io/projected/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-api-access-r9ffs\") pod \"15054085-b6ab-4c89-9db8-dd3ceebe26da\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") "
Jul 01 19:46:43.910670 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910477    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-cabundle-cert\") pod \"15054085-b6ab-4c89-9db8-dd3ceebe26da\" (UID: \"15054085-b6ab-4c89-9db8-dd3ceebe26da\") "
Jul 01 19:46:43.910825 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910724    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15054085-b6ab-4c89-9db8-dd3ceebe26da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15054085-b6ab-4c89-9db8-dd3ceebe26da" (UID: "15054085-b6ab-4c89-9db8-dd3ceebe26da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:46:43.910939 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910894    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15054085-b6ab-4c89-9db8-dd3ceebe26da" (UID: "15054085-b6ab-4c89-9db8-dd3ceebe26da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:46:43.911185 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.910970    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15054085-b6ab-4c89-9db8-dd3ceebe26da" (UID: "15054085-b6ab-4c89-9db8-dd3ceebe26da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:46:43.912658 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.912632    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-api-access-r9ffs" (OuterVolumeSpecName: "kube-api-access-r9ffs") pod "15054085-b6ab-4c89-9db8-dd3ceebe26da" (UID: "15054085-b6ab-4c89-9db8-dd3ceebe26da"). InnerVolumeSpecName "kube-api-access-r9ffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:46:43.912658 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:43.912648    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15054085-b6ab-4c89-9db8-dd3ceebe26da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15054085-b6ab-4c89-9db8-dd3ceebe26da" (UID: "15054085-b6ab-4c89-9db8-dd3ceebe26da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:46:44.011976 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.011909    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:46:44.011976 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.011969    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15054085-b6ab-4c89-9db8-dd3ceebe26da-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:46:44.011976 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.011987    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9ffs\" (UniqueName: \"kubernetes.io/projected/15054085-b6ab-4c89-9db8-dd3ceebe26da-kube-api-access-r9ffs\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:46:44.012211 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.012000    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15054085-b6ab-4c89-9db8-dd3ceebe26da-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:46:44.012211 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.012012    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15054085-b6ab-4c89-9db8-dd3ceebe26da-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:46:44.269771 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.269731    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"]
Jul 01 19:46:44.270033 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270020    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kube-rbac-proxy"
Jul 01 19:46:44.270088 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270035    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kube-rbac-proxy"
Jul 01 19:46:44.270088 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270045    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container"
Jul 01 19:46:44.270088 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270052    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container"
Jul 01 19:46:44.270088 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270060    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="storage-initializer"
Jul 01 19:46:44.270088 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270066    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="storage-initializer"
Jul 01 19:46:44.270241 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270109    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kube-rbac-proxy"
Jul 01 19:46:44.270241 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.270116    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerName="kserve-container"
Jul 01 19:46:44.273380 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.273360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.276126 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.276108    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-68197-kube-rbac-proxy-sar-config\""
Jul 01 19:46:44.276951 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.276933    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-68197-predictor-serving-cert\""
Jul 01 19:46:44.285586 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.285565    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"]
Jul 01 19:46:44.320971 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.320938    2576 generic.go:356] "Generic (PLEG): container finished" podID="15054085-b6ab-4c89-9db8-dd3ceebe26da" containerID="7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655" exitCode=0
Jul 01 19:46:44.321089 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.320993    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"
Jul 01 19:46:44.321089 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.321005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerDied","Data":"7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655"}
Jul 01 19:46:44.321089 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.321067    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz" event={"ID":"15054085-b6ab-4c89-9db8-dd3ceebe26da","Type":"ContainerDied","Data":"85fb7f3214c0d0c99130e6f23145373fa2208491c4f419caf4db8db0c4e28590"}
Jul 01 19:46:44.321244 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.321135    2576 scope.go:117] "RemoveContainer" containerID="1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af"
Jul 01 19:46:44.328871 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.328731    2576 scope.go:117] "RemoveContainer" containerID="7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655"
Jul 01 19:46:44.336320 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.336303    2576 scope.go:117] "RemoveContainer" containerID="27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca"
Jul 01 19:46:44.343035 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.343016    2576 scope.go:117] "RemoveContainer" containerID="1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af"
Jul 01 19:46:44.343324 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:46:44.343303    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af\": container with ID starting with 1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af not found: ID does not exist" containerID="1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af"
Jul 01 19:46:44.343591 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.343333    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af"} err="failed to get container status \"1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af\": rpc error: code = NotFound desc = could not find container \"1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af\": container with ID starting with 1979c879fe51bb8816cb10922e1e167701836a75157612ca4a47d25c38f0f0af not found: ID does not exist"
Jul 01 19:46:44.343591 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.343351    2576 scope.go:117] "RemoveContainer" containerID="7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655"
Jul 01 19:46:44.344264 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:46:44.344243    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655\": container with ID starting with 7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655 not found: ID does not exist" containerID="7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655"
Jul 01 19:46:44.344337 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.344270    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655"} err="failed to get container status \"7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655\": rpc error: code = NotFound desc = could not find container \"7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655\": container with ID starting with 7c2d679dd2ffb6c9307de62b72023eb8744616b1df77fb29ca67add01d623655 not found: ID does not exist"
Jul 01 19:46:44.344337 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.344287    2576 scope.go:117] "RemoveContainer" containerID="27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca"
Jul 01 19:46:44.344612 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:46:44.344595    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca\": container with ID starting with 27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca not found: ID does not exist" containerID="27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca"
Jul 01 19:46:44.344668 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.344617    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca"} err="failed to get container status \"27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca\": rpc error: code = NotFound desc = could not find container \"27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca\": container with ID starting with 27ed3561ea963cb35d317707020e7c252d122af11116d2ca4014a6a1618cc5ca not found: ID does not exist"
Jul 01 19:46:44.345469 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.345451    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"]
Jul 01 19:46:44.351203 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.351179    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz"]
Jul 01 19:46:44.415231 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.415195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.415394 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.415246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-cabundle-cert\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.415394 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.415304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-proxy-tls\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.415394 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.415376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhxb\" (UniqueName: \"kubernetes.io/projected/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-api-access-skhxb\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.415527 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.415405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kserve-provision-location\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.516748 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.516715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhxb\" (UniqueName: \"kubernetes.io/projected/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-api-access-skhxb\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.516748 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.516750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kserve-provision-location\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.516996 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.516769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.516996 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.516790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-cabundle-cert\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.516996 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.516820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-proxy-tls\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.517198 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.517172    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kserve-provision-location\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.517542 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.517525    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-cabundle-cert\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.517598 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.517540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.519308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.519289    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-proxy-tls\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.526987 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.526935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhxb\" (UniqueName: \"kubernetes.io/projected/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-api-access-skhxb\") pod \"raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.582660 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.582629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:44.716398 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.716368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"]
Jul 01 19:46:44.719603 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:46:44.719571    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4b8666_ff4f_4826_ac53_8d68c267dd7c.slice/crio-b8f0daa5ae49dcc02848fbdb99e54f2fe46f4999c0cf46fe66ee1042f3240fed WatchSource:0}: Error finding container b8f0daa5ae49dcc02848fbdb99e54f2fe46f4999c0cf46fe66ee1042f3240fed: Status 404 returned error can't find the container with id b8f0daa5ae49dcc02848fbdb99e54f2fe46f4999c0cf46fe66ee1042f3240fed
Jul 01 19:46:44.794105 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:44.794020    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15054085-b6ab-4c89-9db8-dd3ceebe26da" path="/var/lib/kubelet/pods/15054085-b6ab-4c89-9db8-dd3ceebe26da/volumes"
Jul 01 19:46:45.327148 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:45.327105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerStarted","Data":"5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a"}
Jul 01 19:46:45.327319 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:45.327177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerStarted","Data":"b8f0daa5ae49dcc02848fbdb99e54f2fe46f4999c0cf46fe66ee1042f3240fed"}
Jul 01 19:46:48.338541 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:48.338504    2576 generic.go:356] "Generic (PLEG): container finished" podID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerID="5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a" exitCode=0
Jul 01 19:46:48.338973 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:48.338578    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerDied","Data":"5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a"}
Jul 01 19:46:49.344423 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:49.344388    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerStarted","Data":"eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f"}
Jul 01 19:46:49.344794 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:49.344460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerStarted","Data":"793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f"}
Jul 01 19:46:49.344794 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:49.344770    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:49.344938 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:49.344906    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:49.346222 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:49.346198    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:46:49.368290 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:49.368248    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podStartSLOduration=1.905536897 podStartE2EDuration="5.368234321s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.462697424s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:46:49.36718998 +0000 UTC m=+1476.980233370" watchObservedRunningTime="2026-07-01 19:46:49.368234321 +0000 UTC m=+1476.981277707"
Jul 01 19:46:50.347976 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:50.347910    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:46:51.351275 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:51.351239    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:46:56.355870 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:56.355840    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:46:56.356393 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:46:56.356353    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:47:06.356549 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:47:06.356508    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:47:16.356296 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:47:16.356258    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:47:26.356706 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:47:26.356669    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:47:36.357263 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:47:36.357179    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:47:46.357151 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:47:46.357114    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:47:56.357040 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:47:56.356997    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:48:04.793871 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:04.793843    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:48:14.360964 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:14.360910    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"]
Jul 01 19:48:14.361498 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:14.361158    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" containerID="cri-o://793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f" gracePeriod=30
Jul 01 19:48:14.361498 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:14.361201    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kube-rbac-proxy" containerID="cri-o://eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f" gracePeriod=30
Jul 01 19:48:14.644308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:14.644218    2576 generic.go:356] "Generic (PLEG): container finished" podID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerID="eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f" exitCode=2
Jul 01 19:48:14.644308 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:14.644297    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerDied","Data":"eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f"}
Jul 01 19:48:14.790481 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:14.790444    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 19:48:16.352500 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:16.352458    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 01 19:48:18.905208 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.905179    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:48:18.953888 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.953856    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-proxy-tls\") pod \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") "
Jul 01 19:48:18.954073 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.953974    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kserve-provision-location\") pod \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") "
Jul 01 19:48:18.954073 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.954005    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-cabundle-cert\") pod \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") "
Jul 01 19:48:18.954073 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.954040    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skhxb\" (UniqueName: \"kubernetes.io/projected/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-api-access-skhxb\") pod \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") "
Jul 01 19:48:18.954236 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.954087    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-rbac-proxy-sar-config\") pod \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\" (UID: \"ed4b8666-ff4f-4826-ac53-8d68c267dd7c\") "
Jul 01 19:48:18.954287 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.954237    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed4b8666-ff4f-4826-ac53-8d68c267dd7c" (UID: "ed4b8666-ff4f-4826-ac53-8d68c267dd7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:48:18.954450 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.954429    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed4b8666-ff4f-4826-ac53-8d68c267dd7c" (UID: "ed4b8666-ff4f-4826-ac53-8d68c267dd7c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:48:18.954560 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.954532    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed4b8666-ff4f-4826-ac53-8d68c267dd7c" (UID: "ed4b8666-ff4f-4826-ac53-8d68c267dd7c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:48:18.956081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.956050    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-api-access-skhxb" (OuterVolumeSpecName: "kube-api-access-skhxb") pod "ed4b8666-ff4f-4826-ac53-8d68c267dd7c" (UID: "ed4b8666-ff4f-4826-ac53-8d68c267dd7c"). InnerVolumeSpecName "kube-api-access-skhxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:48:18.956081 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:18.956073    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed4b8666-ff4f-4826-ac53-8d68c267dd7c" (UID: "ed4b8666-ff4f-4826-ac53-8d68c267dd7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:48:19.055609 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.055577    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kserve-provision-location\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:48:19.055609 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.055608    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-cabundle-cert\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:48:19.055805 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.055619    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skhxb\" (UniqueName: \"kubernetes.io/projected/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-api-access-skhxb\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:48:19.055805 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.055628    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:48:19.055805 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.055638    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4b8666-ff4f-4826-ac53-8d68c267dd7c-proxy-tls\") on node \"ip-10-0-143-50.ec2.internal\" DevicePath \"\""
Jul 01 19:48:19.665983 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.665951    2576 generic.go:356] "Generic (PLEG): container finished" podID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerID="793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f" exitCode=0
Jul 01 19:48:19.666167 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.666065    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"
Jul 01 19:48:19.666167 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.666069    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerDied","Data":"793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f"}
Jul 01 19:48:19.666167 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.666134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls" event={"ID":"ed4b8666-ff4f-4826-ac53-8d68c267dd7c","Type":"ContainerDied","Data":"b8f0daa5ae49dcc02848fbdb99e54f2fe46f4999c0cf46fe66ee1042f3240fed"}
Jul 01 19:48:19.666291 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.666173    2576 scope.go:117] "RemoveContainer" containerID="eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f"
Jul 01 19:48:19.674791 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.674761    2576 scope.go:117] "RemoveContainer" containerID="793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f"
Jul 01 19:48:19.681902 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.681883    2576 scope.go:117] "RemoveContainer" containerID="5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a"
Jul 01 19:48:19.689030 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.689010    2576 scope.go:117] "RemoveContainer" containerID="eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f"
Jul 01 19:48:19.689263 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:48:19.689243    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f\": container with ID starting with eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f not found: ID does not exist" containerID="eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f"
Jul 01 19:48:19.689328 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.689276    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f"} err="failed to get container status \"eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f\": rpc error: code = NotFound desc = could not find container \"eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f\": container with ID starting with eedc2c1f40790596539d5a9ec6983f14a728ba0c9f6dd093a081343170a8547f not found: ID does not exist"
Jul 01 19:48:19.689328 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.689300    2576 scope.go:117] "RemoveContainer" containerID="793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f"
Jul 01 19:48:19.689541 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:48:19.689524    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f\": container with ID starting with 793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f not found: ID does not exist" containerID="793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f"
Jul 01 19:48:19.689599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.689545    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f"} err="failed to get container status \"793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f\": rpc error: code = NotFound desc = could not find container \"793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f\": container with ID starting with 793961f3bf9d6994b347fdfa72bf6b74e69f898bc963a850a703ad79c9d51c2f not found: ID does not exist"
Jul 01 19:48:19.689599 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.689559    2576 scope.go:117] "RemoveContainer" containerID="5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a"
Jul 01 19:48:19.689808 ip-10-0-143-50 kubenswrapper[2576]: E0701 19:48:19.689793    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a\": container with ID starting with 5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a not found: ID does not exist" containerID="5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a"
Jul 01 19:48:19.689851 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.689811    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a"} err="failed to get container status \"5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a\": rpc error: code = NotFound desc = could not find container \"5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a\": container with ID starting with 5c035fafff6dc870e07976f81d83083f885f8eda7224a640c3e0072df5b75a6a not found: ID does not exist"
Jul 01 19:48:19.690433 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.690415    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"]
Jul 01 19:48:19.695407 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:19.695389    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls"]
Jul 01 19:48:20.793419 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:20.793389    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" path="/var/lib/kubelet/pods/ed4b8666-ff4f-4826-ac53-8d68c267dd7c/volumes"
Jul 01 19:48:47.589428 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589393    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-726bc/must-gather-dv5xl"]
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589669    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="storage-initializer"
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589680    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="storage-initializer"
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589688    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kube-rbac-proxy"
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589693    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kube-rbac-proxy"
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589701    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container"
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589706    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container"
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589752    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kserve-container"
Jul 01 19:48:47.589894 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.589761    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4b8666-ff4f-4826-ac53-8d68c267dd7c" containerName="kube-rbac-proxy"
Jul 01 19:48:47.592417 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.592400    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:47.595021 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.594989    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-726bc\"/\"kube-root-ca.crt\""
Jul 01 19:48:47.596082 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.596066    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-726bc\"/\"default-dockercfg-rqmdg\""
Jul 01 19:48:47.596136 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.596078    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-726bc\"/\"openshift-service-ca.crt\""
Jul 01 19:48:47.601490 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.601465    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-726bc/must-gather-dv5xl"]
Jul 01 19:48:47.675974 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.675940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgc8g\" (UniqueName: \"kubernetes.io/projected/7959db77-356c-4b0f-a94a-0a3a6e7619c7-kube-api-access-jgc8g\") pod \"must-gather-dv5xl\" (UID: \"7959db77-356c-4b0f-a94a-0a3a6e7619c7\") " pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:47.676127 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.675985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7959db77-356c-4b0f-a94a-0a3a6e7619c7-must-gather-output\") pod \"must-gather-dv5xl\" (UID: \"7959db77-356c-4b0f-a94a-0a3a6e7619c7\") " pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:47.777199 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.777169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgc8g\" (UniqueName: \"kubernetes.io/projected/7959db77-356c-4b0f-a94a-0a3a6e7619c7-kube-api-access-jgc8g\") pod \"must-gather-dv5xl\" (UID: \"7959db77-356c-4b0f-a94a-0a3a6e7619c7\") " pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:47.777358 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.777214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7959db77-356c-4b0f-a94a-0a3a6e7619c7-must-gather-output\") pod \"must-gather-dv5xl\" (UID: \"7959db77-356c-4b0f-a94a-0a3a6e7619c7\") " pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:47.777512 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.777496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7959db77-356c-4b0f-a94a-0a3a6e7619c7-must-gather-output\") pod \"must-gather-dv5xl\" (UID: \"7959db77-356c-4b0f-a94a-0a3a6e7619c7\") " pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:47.786661 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.786636    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgc8g\" (UniqueName: \"kubernetes.io/projected/7959db77-356c-4b0f-a94a-0a3a6e7619c7-kube-api-access-jgc8g\") pod \"must-gather-dv5xl\" (UID: \"7959db77-356c-4b0f-a94a-0a3a6e7619c7\") " pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:47.901173 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:47.901091    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-726bc/must-gather-dv5xl"
Jul 01 19:48:48.022811 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:48.022787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-726bc/must-gather-dv5xl"]
Jul 01 19:48:48.025372 ip-10-0-143-50 kubenswrapper[2576]: W0701 19:48:48.025339    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7959db77_356c_4b0f_a94a_0a3a6e7619c7.slice/crio-54d3cc1653e1222a969b1ed3c1cac61d964fa07eae68aae77943f75b94297120 WatchSource:0}: Error finding container 54d3cc1653e1222a969b1ed3c1cac61d964fa07eae68aae77943f75b94297120: Status 404 returned error can't find the container with id 54d3cc1653e1222a969b1ed3c1cac61d964fa07eae68aae77943f75b94297120
Jul 01 19:48:48.027267 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:48.027248    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:48:48.768699 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:48.768660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-726bc/must-gather-dv5xl" event={"ID":"7959db77-356c-4b0f-a94a-0a3a6e7619c7","Type":"ContainerStarted","Data":"54d3cc1653e1222a969b1ed3c1cac61d964fa07eae68aae77943f75b94297120"}
Jul 01 19:48:49.774629 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:49.774599    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-726bc/must-gather-dv5xl" event={"ID":"7959db77-356c-4b0f-a94a-0a3a6e7619c7","Type":"ContainerStarted","Data":"3db511d20b4454b02de47fd3d7932ee2567fea2fd5a8234911c8259469a5be54"}
Jul 01 19:48:49.775028 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:49.774639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-726bc/must-gather-dv5xl" event={"ID":"7959db77-356c-4b0f-a94a-0a3a6e7619c7","Type":"ContainerStarted","Data":"1e5d73314133af6db30398dfdc2f8878115a785243a95e97c24a7829cccbf7ae"}
Jul 01 19:48:49.795007 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:49.793836    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-726bc/must-gather-dv5xl" podStartSLOduration=2.078674041 podStartE2EDuration="2.793815946s" totalImagesPullingTime="715.141905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:48:49.79269153 +0000 UTC m=+1597.405734930" watchObservedRunningTime="2026-07-01 19:48:49.793815946 +0000 UTC m=+1597.406859333"
Jul 01 19:48:50.255316 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:50.255293    2576 ???:1] "http: TLS handshake error from 10.0.142.179:52384: EOF"
Jul 01 19:48:50.260125 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:50.260076    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2kwwj_3a87f684-4294-4618-ba06-c37040b80606/global-pull-secret-syncer/0.log"
Jul 01 19:48:50.499730 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:50.499701    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pdkcn_4b2bb51a-496e-4929-b481-922ce49e4767/konnectivity-agent/0.log"
Jul 01 19:48:50.575624 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:50.575541    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-50.ec2.internal_ef75bb6a2b1f0b788d8d4b4eec583ca4/haproxy/0.log"
Jul 01 19:48:54.173211 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:54.173176    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r6jz_24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e/node-exporter/0.log"
Jul 01 19:48:54.215979 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:54.215942    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r6jz_24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e/kube-rbac-proxy/0.log"
Jul 01 19:48:54.247362 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:54.247326    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r6jz_24b64bcb-2e76-48f8-bdb8-e1df1f6dd62e/init-textfile/0.log"
Jul 01 19:48:56.668797 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:56.668764    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-kqc77_a110deee-26de-412e-a746-2bbd075dd8c2/networking-console-plugin/0.log"
Jul 01 19:48:57.758708 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:57.758679    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-vdvph_a59f23c7-910a-47a4-8350-067355cc0aec/download-server/0.log"
Jul 01 19:48:58.487625 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.487590    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"]
Jul 01 19:48:58.493033 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.492999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.504541 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.504511    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"]
Jul 01 19:48:58.675436 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.675403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cwh\" (UniqueName: \"kubernetes.io/projected/f619416c-798e-433f-bf44-b257cd6a74c3-kube-api-access-c8cwh\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.675615 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.675464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-podres\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.675615 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.675542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-sys\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.675615 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.675575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-proc\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.675615 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.675590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-lib-modules\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.776571 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.776529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-sys\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777194 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.776813    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-sys\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777384 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.777173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-proc\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777482 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.777419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-lib-modules\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777482 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.777250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-proc\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777482 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.777469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cwh\" (UniqueName: \"kubernetes.io/projected/f619416c-798e-433f-bf44-b257cd6a74c3-kube-api-access-c8cwh\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777605 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.777535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-podres\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777655 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.777609    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-lib-modules\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.777695 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.777668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f619416c-798e-433f-bf44-b257cd6a74c3-podres\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.795087 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.794938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cwh\" (UniqueName: \"kubernetes.io/projected/f619416c-798e-433f-bf44-b257cd6a74c3-kube-api-access-c8cwh\") pod \"perf-node-gather-daemonset-n6t4w\" (UID: \"f619416c-798e-433f-bf44-b257cd6a74c3\") " pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.804270 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.804247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:58.973590 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:58.973559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"]
Jul 01 19:48:59.495950 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:59.495870    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vnkct_628e1705-7c95-42b2-a3fd-c524a324c87a/dns/0.log"
Jul 01 19:48:59.557204 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:59.557177    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vnkct_628e1705-7c95-42b2-a3fd-c524a324c87a/kube-rbac-proxy/0.log"
Jul 01 19:48:59.590931 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:59.590894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7ktbn_3023be20-3d3a-4833-914f-1f98ea4ba5ee/dns-node-resolver/0.log"
Jul 01 19:48:59.831403 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:59.831372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w" event={"ID":"f619416c-798e-433f-bf44-b257cd6a74c3","Type":"ContainerStarted","Data":"854d7818645613d805c033d3e4601b4aca901caa6ecfef8438dee796541d6058"}
Jul 01 19:48:59.831809 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:59.831423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w" event={"ID":"f619416c-798e-433f-bf44-b257cd6a74c3","Type":"ContainerStarted","Data":"bf58e8dff34d00abd447a7354047dfc6a7c832d04ba10a3526d873fe17125efa"}
Jul 01 19:48:59.831809 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:59.831522    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:48:59.854421 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:48:59.854364    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w" podStartSLOduration=1.854347315 podStartE2EDuration="1.854347315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:48:59.853799285 +0000 UTC m=+1607.466842674" watchObservedRunningTime="2026-07-01 19:48:59.854347315 +0000 UTC m=+1607.467390701"
Jul 01 19:49:00.181904 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:00.181824    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cg7l_b755486f-bcce-43d8-b6d7-c06e780555fe/node-ca/0.log"
Jul 01 19:49:01.621676 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:01.621647    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5mdvp_37f7468b-37a0-4ab3-b03b-6ff4792847c6/serve-healthcheck-canary/0.log"
Jul 01 19:49:02.081601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:02.081574    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dgnqs_a7fdf84c-9fdd-43e8-8200-cb82e5b73262/kube-rbac-proxy/0.log"
Jul 01 19:49:02.107391 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:02.107370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dgnqs_a7fdf84c-9fdd-43e8-8200-cb82e5b73262/exporter/0.log"
Jul 01 19:49:02.136324 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:02.136302    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dgnqs_a7fdf84c-9fdd-43e8-8200-cb82e5b73262/extractor/0.log"
Jul 01 19:49:04.608881 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:04.608793    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-q6v7c_c62add65-1abd-4907-826b-dd66dfa99917/manager/0.log"
Jul 01 19:49:05.846264 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:05.846237    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-726bc/perf-node-gather-daemonset-n6t4w"
Jul 01 19:49:09.377601 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:09.377564    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-jjrkw_e2a85bfa-282d-4472-94ca-4811bb44335b/kube-storage-version-migrator-operator/1.log"
Jul 01 19:49:09.378651 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:09.378631    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-jjrkw_e2a85bfa-282d-4472-94ca-4811bb44335b/kube-storage-version-migrator-operator/0.log"
Jul 01 19:49:10.430780 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:10.430751    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xg25_256697b6-5404-4d73-94a9-d2363e8c37f1/kube-multus-additional-cni-plugins/0.log"
Jul 01 19:49:10.457386 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:10.457361    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xg25_256697b6-5404-4d73-94a9-d2363e8c37f1/egress-router-binary-copy/0.log"
Jul 01 19:49:10.481730 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:10.481702    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xg25_256697b6-5404-4d73-94a9-d2363e8c37f1/cni-plugins/0.log"
Jul 01 19:49:10.507432 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:10.507404    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xg25_256697b6-5404-4d73-94a9-d2363e8c37f1/bond-cni-plugin/0.log"
Jul 01 19:49:10.534621 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:10.534595    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xg25_256697b6-5404-4d73-94a9-d2363e8c37f1/routeoverride-cni/0.log"
Jul 01 19:49:10.560977 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:10.560892    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xg25_256697b6-5404-4d73-94a9-d2363e8c37f1/whereabouts-cni-bincopy/0.log"
Jul 01 19:49:10.583815 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:10.583783    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xg25_256697b6-5404-4d73-94a9-d2363e8c37f1/whereabouts-cni/0.log"
Jul 01 19:49:11.056617 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:11.056576    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t2nzh_247b7370-a4e2-4213-a816-91ca08794aae/kube-multus/0.log"
Jul 01 19:49:11.079976 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:11.079951    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6btbd_59d8dc7f-ebf5-41cb-8951-339301976b04/network-metrics-daemon/0.log"
Jul 01 19:49:11.105865 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:11.105841    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6btbd_59d8dc7f-ebf5-41cb-8951-339301976b04/kube-rbac-proxy/0.log"
Jul 01 19:49:12.003382 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.003356    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/ovn-controller/0.log"
Jul 01 19:49:12.023361 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.023339    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/ovn-acl-logging/0.log"
Jul 01 19:49:12.049959 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.049937    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/kube-rbac-proxy-node/0.log"
Jul 01 19:49:12.075692 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.075663    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 19:49:12.108169 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.108145    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/northd/0.log"
Jul 01 19:49:12.136524 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.136498    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/nbdb/0.log"
Jul 01 19:49:12.163465 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.163433    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/sbdb/0.log"
Jul 01 19:49:12.215861 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:12.215832    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4ppkh_bc73ea4b-d504-4d79-be60-6eedf337dc0a/ovnkube-controller/0.log"
Jul 01 19:49:13.823833 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:13.823799    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-p6b5k_60335b20-da5c-44f7-8ef1-e1c9858fbff9/network-check-target-container/0.log"
Jul 01 19:49:14.837188 ip-10-0-143-50 kubenswrapper[2576]: I0701 19:49:14.837162    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-hgqkp_af3076a6-12bd-405c-8ec3-d1fce98fdb9a/iptables-alerter/0.log"