Jun 15 05:47:41.535140 ip-10-0-135-167 systemd[1]: Starting Kubernetes Kubelet...
Jun 15 05:47:42.000412 ip-10-0-135-167 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:42.000412 ip-10-0-135-167 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 15 05:47:42.000412 ip-10-0-135-167 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:42.000412 ip-10-0-135-167 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 15 05:47:42.000412 ip-10-0-135-167 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:42.003285 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.003191    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 15 05:47:42.008458 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008436    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:42.008458 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008454    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:42.008458 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008460    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:42.008458 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008464    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008468    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008473    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008477    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008481    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008485    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008489    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008493    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008497    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008501    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008518    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008523    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008527    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008531    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008535    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008540    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008544    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008550    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008554    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008558    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:42.008711 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008562    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008566    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008571    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008574    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008579    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008583    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008587    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008591    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008595    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008600    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008604    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008608    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008612    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008617    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008622    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008628    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008632    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008637    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008641    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008646    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:42.009540 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008650    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008655    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008659    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008663    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008670    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008676    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008681    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008686    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008691    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008696    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008701    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008705    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008709    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008713    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008717    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008737    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008743    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008747    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008751    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008755    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:42.010503 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008759    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008764    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008769    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008773    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008780    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008786    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008792    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008797    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008803    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008808    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008813    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008818    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008822    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008828    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008833    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008837    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008844    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008850    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008854    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:42.011336 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008859    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008864    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008868    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008872    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008876    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008881    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008885    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008889    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008893    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008898    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008903    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008909    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008914    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008918    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008923    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008927    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.008931    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010136    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010148    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010154    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:42.012212 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010160    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010164    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010170    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010176    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010180    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010185    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010190    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010194    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010198    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010203    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010207    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010211    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010216    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010220    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010224    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010229    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010236    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010242    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010247    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:42.013064 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010251    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010256    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010260    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010264    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010268    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010273    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010279    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010285    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010289    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010294    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010298    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010302    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010308    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010313    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010317    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010321    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010326    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010331    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010336    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010340    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:42.013637 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010344    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010349    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010353    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010357    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010362    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010365    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010370    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010374    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010379    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010383    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010387    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010391    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010396    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010400    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010404    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010408    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010412    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010416    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010420    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010424    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:42.014227 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010428    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010435    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010440    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010445    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010451    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010456    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010460    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010464    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010470    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010475    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010479    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010484    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010488    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010492    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010496    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010500    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010504    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010508    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010512    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010516    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:42.014847 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010520    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010524    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010528    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010533    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010537    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010541    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010547    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010552    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010556    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010561    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010565    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010569    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010573    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010577    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010581    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010585    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.010590    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011218    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011227    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:42.015461 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011232    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011236    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011241    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011245    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011250    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011254    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011258    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011263    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011267    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011271    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011275    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011279    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011283    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011288    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011292    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011296    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011300    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011304    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011311    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:42.015958 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011317    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011322    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011326    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011331    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011335    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011339    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011344    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011348    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011351    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011355    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011360    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011365    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011369    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011373    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011378    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011382    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011387    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011394    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011400    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:42.016522 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011404    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011410    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011415    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011419    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011424    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011428    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011432    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011437    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011441    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011445    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011449    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011453    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011457    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011462    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011466    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011472    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011478    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011483    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011487    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:42.017013 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011491    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011496    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011500    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011504    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011510    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011514    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011518    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011523    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011527    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011531    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011535    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011540    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011545    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011549    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011553    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011558    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011563    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011567    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011571    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011575    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:42.017479 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011578    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011583    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011587    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011591    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011595    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011599    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011604    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011608    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011613    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011617    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011621    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011625    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011629    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011633    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011637    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011641    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011646    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011651    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011655    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.011659    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:42.018017 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011782    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011794    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011805    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011812    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011819    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011824    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011831    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011838    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011844    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011848    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011854    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011859    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011864    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011869    2579 flags.go:64] FLAG: --cgroup-root=""
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011873    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011878    2579 flags.go:64] FLAG: --client-ca-file=""
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011883    2579 flags.go:64] FLAG: --cloud-provider="external"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011888    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011900    2579 flags.go:64] FLAG: --cluster-domain=""
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011906    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011911    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011917    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011922    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011929    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 15 05:47:42.018494 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011935    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011940    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011946    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011950    2579 flags.go:64] FLAG: --contention-profiling="false"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011956    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011961    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011966    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011971    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011978    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011983    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011988    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011993    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.011997    2579 flags.go:64] FLAG: --enable-server="true"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012003    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012019    2579 flags.go:64] FLAG: --event-burst="100"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012025    2579 flags.go:64] FLAG: --event-qps="50"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012029    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012034    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012039    2579 flags.go:64] FLAG: --eviction-hard=""
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012051    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012057    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012065    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012070    2579 flags.go:64] FLAG: --eviction-soft=""
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012075    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012080    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 15 05:47:42.019114 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012085    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012090    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012095    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012099    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012104    2579 flags.go:64] FLAG: --feature-gates=""
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012111    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012115    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012120    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012125    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012131    2579 flags.go:64] FLAG: --healthz-port="10248"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012136    2579 flags.go:64] FLAG: --help="false"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012141    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012146    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012151    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012156    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012161    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012167    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012173    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012178    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012184    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012189    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012195    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012200    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012204    2579 flags.go:64] FLAG: --kube-reserved=""
Jun 15 05:47:42.019874 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012209    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012213    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012218    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012222    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012232    2579 flags.go:64] FLAG: --lock-file=""
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012236    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012241    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012246    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012256    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012261    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012266    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012270    2579 flags.go:64] FLAG: --logging-format="text"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012275    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012280    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012285    2579 flags.go:64] FLAG: --manifest-url=""
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012289    2579 flags.go:64] FLAG: --manifest-url-header=""
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012297    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012302    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012309    2579 flags.go:64] FLAG: --max-pods="110"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012317    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012322    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012326    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012332    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012336    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012341    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 15 05:47:42.020928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012346    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012357    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012362    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012367    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012372    2579 flags.go:64] FLAG: --pod-cidr=""
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012378    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a2e90c8ef7d783e4d635e24fb09435f5e1f6e954583512c0944f9e57bc77b52"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012387    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012392    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012397    2579 flags.go:64] FLAG: --pods-per-core="0"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012402    2579 flags.go:64] FLAG: --port="10250"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012407    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012414    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a9c1276393efef28"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012419    2579 flags.go:64] FLAG: --qos-reserved=""
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012425    2579 flags.go:64] FLAG: --read-only-port="10255"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012430    2579 flags.go:64] FLAG: --register-node="true"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012435    2579 flags.go:64] FLAG: --register-with-taints=""
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012441    2579 flags.go:64] FLAG: --registry-burst="10"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012445    2579 flags.go:64] FLAG: --registry-qps="5"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012450    2579 flags.go:64] FLAG: --reserved-cpus=""
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012455    2579 flags.go:64] FLAG: --reserved-memory=""
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012462    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012467    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012472    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012476    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012481    2579 flags.go:64] FLAG: --runonce="false"
Jun 15 05:47:42.021645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012485    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012490    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012498    2579 flags.go:64] FLAG: --seccomp-default="false"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012502    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012508    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012513    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012518    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012523    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012527    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012532    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012537    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012541    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012547    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012552    2579 flags.go:64] FLAG: --system-cgroups=""
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012557    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012565    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012570    2579 flags.go:64] FLAG: --tls-cert-file=""
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012575    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012582    2579 flags.go:64] FLAG: --tls-min-version=""
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012587    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012592    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012596    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012601    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012606    2579 flags.go:64] FLAG: --v="2"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012612    2579 flags.go:64] FLAG: --version="false"
Jun 15 05:47:42.022308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012619    2579 flags.go:64] FLAG: --vmodule=""
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012625    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.012631    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012834    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012841    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012846    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012850    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012854    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012858    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012865    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012869    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012874    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012878    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012882    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012889    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012895    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012899    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012903    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012907    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012912    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:42.023061 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012916    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012922    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012926    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012930    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012934    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012940    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012944    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012948    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012953    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012957    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012961    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012966    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012970    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012974    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012980    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012986    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012991    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.012996    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013000    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013004    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:42.023535 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013008    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013014    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013019    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013023    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013027    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013031    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013035    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013039    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013043    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013048    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013052    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013056    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013060    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013064    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013069    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013072    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013076    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013082    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013086    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:42.024056 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013091    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013097    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013103    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013108    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013113    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013117    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013122    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013126    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013130    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013134    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013139    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013143    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013148    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013152    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013158    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013162    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013166    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013170    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013174    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:42.024511 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013179    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013183    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013187    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013191    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013195    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013200    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013204    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013208    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013212    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013217    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013221    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013227    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013231    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013235    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013240    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013244    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013248    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013252    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013256    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013260    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:42.025002 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013265    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013270    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013274    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.013278    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.013951    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.022229    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.022245    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.023047    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.023056    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023116    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023121    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023125    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023128    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023131    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:42.025476 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023136    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023140    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023143    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023146    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023148    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023151    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023154    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023157    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023160    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023162    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023165    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023167    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023171    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023175    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023177    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023180    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023183    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023186    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023188    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:42.025849 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023191    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023194    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023197    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023200    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023203    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023207    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023209    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023212    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023215    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023218    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023222    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023226    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023229    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023232    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023234    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023237    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023241    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023244    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023247    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023249    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:42.026315 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023252    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023254    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023257    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023259    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023262    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023264    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023267    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023269    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023272    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023274    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023277    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023280    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023282    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023285    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023287    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023290    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023293    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023297    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023300    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023302    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:42.026869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023305    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023308    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023310    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023313    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023315    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023318    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023320    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023323    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023326    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023329    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023331    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023334    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023336    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023339    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023341    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023345    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023347    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023350    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023352    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023355    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:42.027374 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023357    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023360    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023362    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023365    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023367    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023370    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023372    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023375    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023377    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023381    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023384    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023386    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023389    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023391    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023394    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:42.027869 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.023399    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023513    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023518    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023522    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023524    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023527    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023531    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023533    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023536    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023539    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023541    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023544    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023547    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023553    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023555    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023558    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023561    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023563    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023566    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023569    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:42.028249 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023572    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023574    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023577    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023580    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023582    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023585    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023587    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023590    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023593    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023595    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023598    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023602    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023606    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023609    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023612    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023615    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023618    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023621    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023624    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:42.028741 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023627    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023631    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023635    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023638    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023640    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023643    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023646    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023649    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023652    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023654    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023657    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023659    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023662    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023665    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023667    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023670    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023672    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023675    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023678    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:42.029210 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023680    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023683    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023686    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023689    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023691    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023694    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023697    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023700    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023702    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023705    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023707    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023710    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023712    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023715    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023717    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023720    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023735    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023738    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023741    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023744    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:42.029689 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023746    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023750    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023752    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023755    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023757    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023760    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023763    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023765    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023768    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023770    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023773    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023776    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023779    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023782    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023786    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023789    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023792    2579 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023795    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023797    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023800    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:42.030192 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023803    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:42.030672 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:42.023805    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:42.030672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.023810    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:42.030672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.023937    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 15 05:47:42.030672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.026574    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 15 05:47:42.030672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.028132    2579 server.go:1019] "Starting client certificate rotation"
Jun 15 05:47:42.030672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.028234    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:47:42.030672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.028265    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:47:42.031452 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.031438    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:47:42.036707 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.036690    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:47:42.046010 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.045990    2579 log.go:25] "Validated CRI v1 runtime API"
Jun 15 05:47:42.048929 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.048910    2579 log.go:25] "Validated CRI v1 image API"
Jun 15 05:47:42.049683 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.049668    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 15 05:47:42.053586 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.053565    2579 fs.go:135] Filesystem UUIDs: map[096bc139-4745-4010-a469-8248506337d4:/dev/nvme0n1p4 6e12de72-c2b7-4a49-a189-141f79029e83:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jun 15 05:47:42.053686 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.053584    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 15 05:47:42.056336 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.056313    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:47:42.060695 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.060579    2579 manager.go:216] Machine: {Timestamp:2026-06-15 05:47:42.058125613 +0000 UTC m=+0.379042891 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101737 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f121fc8bbfd55f9d406ed3267e485 SystemUUID:ec2f121f-c8bb-fd55-f9d4-06ed3267e485 BootID:53a5fc30-f411-44b5-b953-d99a6e3f1d96 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:18:42:2c:ad:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:18:42:2c:ad:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:79:03:e3:60:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 15 05:47:42.060695 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.060690    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 15 05:47:42.060838 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.060784    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.120.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260608-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 15 05:47:42.062707 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.062644    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 15 05:47:42.063127 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.062744    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-167.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 15 05:47:42.063954 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.063941    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jun 15 05:47:42.063992 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.063958    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jun 15 05:47:42.063992 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.063966    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:47:42.064595 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.064585    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:47:42.065967 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.065957    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jun 15 05:47:42.068622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.068611    2579 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:47:42.068755 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.068745    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jun 15 05:47:42.068790 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.068776    2579 kubelet.go:486] "Attempting to sync node with API server"
Jun 15 05:47:42.068790 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.068785    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 15 05:47:42.068850 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.068795    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 15 05:47:42.068850 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.068805    2579 kubelet.go:398] "Adding apiserver pod source"
Jun 15 05:47:42.068850 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.068814    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 15 05:47:42.069221 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.069208    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:47:42.069270 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.069232    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:47:42.072058 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.072038    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jun 15 05:47:42.073542 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073529    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 15 05:47:42.073600 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073560    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jun 15 05:47:42.073772 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073761    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 15 05:47:42.073804 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073777    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 15 05:47:42.073804 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073783    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 15 05:47:42.073804 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073789    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 15 05:47:42.073804 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073797    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 15 05:47:42.073804 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.073803    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 15 05:47:42.074341 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.074333    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 15 05:47:42.074371 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.074342    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 15 05:47:42.074371 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.074348    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 15 05:47:42.074371 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.074355    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 15 05:47:42.074371 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.074363    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 15 05:47:42.074371 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.074371    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 15 05:47:42.075740 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.075714    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 15 05:47:42.075814 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.075743    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 15 05:47:42.079145 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.079117    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 15 05:47:42.079239 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.079165    2579 server.go:1268] "Started kubelet"
Jun 15 05:47:42.079307 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.079281    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 15 05:47:42.079404 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.079332    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 15 05:47:42.079458 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.079428    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 15 05:47:42.079493 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.079441    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jun 15 05:47:42.079604 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.079591    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-167.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 15 05:47:42.081466 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.081446    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 15 05:47:42.081658 ip-10-0-135-167 systemd[1]: Started Kubernetes Kubelet.
Jun 15 05:47:42.082822 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.082803    2579 server.go:310] "Adding debug handlers to kubelet server"
Jun 15 05:47:42.086021 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.085026    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18b92b0ceab60bcd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-06-15 05:47:42.079142861 +0000 UTC m=+0.400060138,LastTimestamp:2026-06-15 05:47:42.079142861 +0000 UTC m=+0.400060138,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Jun 15 05:47:42.087031 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.087012    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 15 05:47:42.087031 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.087022    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 15 05:47:42.087148 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.087089    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.087983 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.087781    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jun 15 05:47:42.087983 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.087787    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jun 15 05:47:42.088136 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.087985    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 15 05:47:42.088136 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.087996    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jun 15 05:47:42.088136 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088102    2579 reconstruct.go:97] "Volume reconstruction finished"
Jun 15 05:47:42.088136 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088113    2579 reconciler.go:29] "Reconciler: start to sync state"
Jun 15 05:47:42.088311 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088198    2579 factory.go:55] Registering systemd factory
Jun 15 05:47:42.088311 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088211    2579 factory.go:223] Registration of the systemd container factory successfully
Jun 15 05:47:42.088505 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088490    2579 factory.go:153] Registering CRI-O factory
Jun 15 05:47:42.088505 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088507    2579 factory.go:223] Registration of the crio container factory successfully
Jun 15 05:47:42.088637 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088610    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 15 05:47:42.088637 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088630    2579 factory.go:103] Registering Raw factory
Jun 15 05:47:42.088743 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.088642    2579 manager.go:1196] Started watching for new ooms in manager
Jun 15 05:47:42.089295 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.089280    2579 manager.go:349] Starting recovery of all containers
Jun 15 05:47:42.097177 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.097142    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jun 15 05:47:42.097354 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.097326    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jun 15 05:47:42.100050 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.100023    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pgt7l"
Jun 15 05:47:42.100887 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.100869    2579 manager.go:354] Recovery completed
Jun 15 05:47:42.102220 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.102202    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jun 15 05:47:42.105119 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.105107    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:42.106910 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.106892    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:42.106989 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.106924    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:42.106989 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.106935    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:42.107277 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.107261    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pgt7l"
Jun 15 05:47:42.107554 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.107536    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 15 05:47:42.107554 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.107552    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 15 05:47:42.107647 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.107574    2579 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:47:42.107647 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.107579    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:42.108394 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.108376    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:42.108471 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.108408    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:42.108471 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.108422    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:42.109238 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.109222    2579 policy_none.go:49] "None policy: Start"
Jun 15 05:47:42.109238 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.109238    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jun 15 05:47:42.109375 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.109249    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jun 15 05:47:42.110329 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.110316    2579 policy_none.go:47] "Start"
Jun 15 05:47:42.142350 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.142333    2579 manager.go:337] "Starting Device Plugin manager"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.142373    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.142387    2579 server.go:87] "Starting device plugin registration server"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.142643    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.142655    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.142758    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.142833    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.142840    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.143407    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.143454    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.152437    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.153618    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.153662    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jun 15 05:47:42.153702 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.153688    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jun 15 05:47:42.154207 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.153744    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 15 05:47:42.156152 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.156134    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:42.243594 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.243553    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:42.244692 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.244663    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:42.244846 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.244701    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:42.244846 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.244716    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:42.244846 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.244760    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.253943 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.253889    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"]
Jun 15 05:47:42.253996 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.253961    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:42.254917 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.254898    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:42.255014 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.254929    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:42.255014 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.254938    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:42.255075 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.255064    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.255108 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.255085    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-167.ec2.internal\": node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.256058 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256045    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:42.256167 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.256251 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256183    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:42.256759 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256740    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:42.256868 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256770    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:42.256868 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256782    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:42.256868 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256739    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:42.256868 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256858    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:42.257020 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.256872    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:42.257916 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.257900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.258019 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.257931    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:42.258572 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.258555    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:42.258650 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.258586    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:42.258650 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.258601    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:42.275698 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.275673    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.288575 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.288550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/385fb380decbfa57906c03876915ce49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"385fb380decbfa57906c03876915ce49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.288575 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.288570    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-167.ec2.internal\" not found" node="ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.288722 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.288592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fb380decbfa57906c03876915ce49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"385fb380decbfa57906c03876915ce49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.288722 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.288623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e3ac6eed3caa2ac33824ee71932a85f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-167.ec2.internal\" (UID: \"9e3ac6eed3caa2ac33824ee71932a85f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.293189 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.293171    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-167.ec2.internal\" not found" node="ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.376507 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.376477    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.388787 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.388755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/385fb380decbfa57906c03876915ce49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"385fb380decbfa57906c03876915ce49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.388948 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.388790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fb380decbfa57906c03876915ce49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"385fb380decbfa57906c03876915ce49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.388948 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.388843    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/385fb380decbfa57906c03876915ce49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"385fb380decbfa57906c03876915ce49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.388948 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.388852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e3ac6eed3caa2ac33824ee71932a85f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-167.ec2.internal\" (UID: \"9e3ac6eed3caa2ac33824ee71932a85f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.388948 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.388861    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/385fb380decbfa57906c03876915ce49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"385fb380decbfa57906c03876915ce49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.388948 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.388877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e3ac6eed3caa2ac33824ee71932a85f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-167.ec2.internal\" (UID: \"9e3ac6eed3caa2ac33824ee71932a85f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.476946 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.476898    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.577683 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.577581    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.591025 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.590996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.597083 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:42.597059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:42.678412 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.678370    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.778994 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.778946    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.879502 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.879422    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:42.980218 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:42.980184    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:43.027706 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.027672    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 15 05:47:43.028648 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.027864    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 15 05:47:43.080272 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:43.080240    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:43.102524 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:43.102489    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385fb380decbfa57906c03876915ce49.slice/crio-93f944a503ec19ca337cf9154fc6b1f6381ef23826d14de2ace22f6c52755242 WatchSource:0}: Error finding container 93f944a503ec19ca337cf9154fc6b1f6381ef23826d14de2ace22f6c52755242: Status 404 returned error can't find the container with id 93f944a503ec19ca337cf9154fc6b1f6381ef23826d14de2ace22f6c52755242
Jun 15 05:47:43.102752 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:43.102715    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3ac6eed3caa2ac33824ee71932a85f.slice/crio-37d4e5d508d33561a4093703d4dbff8c7c0f8924190e53940e4d69b0b09793f3 WatchSource:0}: Error finding container 37d4e5d508d33561a4093703d4dbff8c7c0f8924190e53940e4d69b0b09793f3: Status 404 returned error can't find the container with id 37d4e5d508d33561a4093703d4dbff8c7c0f8924190e53940e4d69b0b09793f3
Jun 15 05:47:43.106610 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.106594    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:47:43.111267 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.111239    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-14 05:42:42 +0000 UTC" deadline="2028-02-08 02:01:42.591290034 +0000 UTC"
Jun 15 05:47:43.111267 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.111265    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14468h13m59.480027886s"
Jun 15 05:47:43.157875 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.157746    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal" event={"ID":"9e3ac6eed3caa2ac33824ee71932a85f","Type":"ContainerStarted","Data":"37d4e5d508d33561a4093703d4dbff8c7c0f8924190e53940e4d69b0b09793f3"}
Jun 15 05:47:43.158781 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.158754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" event={"ID":"385fb380decbfa57906c03876915ce49","Type":"ContainerStarted","Data":"93f944a503ec19ca337cf9154fc6b1f6381ef23826d14de2ace22f6c52755242"}
Jun 15 05:47:43.181012 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:43.180965    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:43.217581 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.217552    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:43.281264 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:43.281224    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:43.381873 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:43.381842    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Jun 15 05:47:43.391198 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.391177    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:43.406250 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.406230    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:43.487842 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.487747    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:43.500741 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.500694    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:47:43.502020 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.501999    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Jun 15 05:47:43.509518 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:43.509472    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:47:44.002970 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.002939    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:44.070903 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.070869    2579 apiserver.go:52] "Watching apiserver"
Jun 15 05:47:44.076614 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.076588    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 15 05:47:44.077017 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.076993    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal","openshift-multus/multus-additional-cni-plugins-822r9","openshift-network-operator/iptables-alerter-kjqsr","openshift-ovn-kubernetes/ovnkube-node-gnn2f","kube-system/global-pull-secret-syncer-7qgzl","kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal","openshift-dns/node-resolver-xcp6n","openshift-multus/multus-lfbdv","openshift-multus/network-metrics-daemon-smsbk","openshift-network-diagnostics/network-check-target-thbgq","kube-system/konnectivity-agent-9c4zl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb","openshift-cluster-node-tuning-operator/tuned-2mvd7","openshift-image-registry/node-ca-2vs2q"]
Jun 15 05:47:44.079499 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.079480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.081574 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.081553    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 15 05:47:44.081716 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.081552    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 15 05:47:44.081802 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.081781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.081864 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.081797    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2z7r6\""
Jun 15 05:47:44.084155 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.083785    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:47:44.084155 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.083864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8sw7j\""
Jun 15 05:47:44.084155 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.083919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.084155 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.083960    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 15 05:47:44.087986 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.086063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-br642\""
Jun 15 05:47:44.087986 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.086292    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 15 05:47:44.087986 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.086502    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:47:44.087986 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.086511    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 15 05:47:44.087986 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.087792    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 15 05:47:44.088470 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.088203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.090011 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.089992    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 15 05:47:44.090383 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.090365    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 15 05:47:44.090510 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.090489    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 15 05:47:44.090614 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.090599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mhrpg\""
Jun 15 05:47:44.091048 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.091028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 15 05:47:44.091214 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.091196    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 15 05:47:44.091320 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.091301    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 15 05:47:44.091644 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.091627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.091755 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.091697    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:44.094062 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.093971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.096108 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.095934    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 15 05:47:44.096108 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.095969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bhg9n\""
Jun 15 05:47:44.096108 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-kubernetes\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.096108 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-systemd\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.096108 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-systemd-units\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-etc-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-cni-netd\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-dbus\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysconfig\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-sys\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096218    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-lib-modules\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-host\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-ovnkube-config\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.096395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/175a2c40-7bff-4021-a86a-524baa098f52-ovn-node-metrics-cert\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31e44b4-6f88-42da-b187-34d230a5ded0-tmp-dir\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-kubelet\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-systemd\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096472    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-log-socket\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-cni-bin\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdl8d\" (UniqueName: \"kubernetes.io/projected/175a2c40-7bff-4021-a86a-524baa098f52-kube-api-access-fdl8d\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.096530    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-modprobe-d\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysctl-d\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-var-lib-kubelet\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c95a18-cd99-4d98-aa17-5334445a567d-tmp\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zm7\" (UniqueName: \"kubernetes.io/projected/78c95a18-cd99-4d98-aa17-5334445a567d-kube-api-access-p7zm7\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bda06406-4b48-4943-9447-16483d800bda-iptables-alerter-script\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-env-overrides\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78c95a18-cd99-4d98-aa17-5334445a567d-etc-tuned\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-run-netns\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysctl-conf\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bda06406-4b48-4943-9447-16483d800bda-host-slash\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8fhq\" (UniqueName: \"kubernetes.io/projected/bda06406-4b48-4943-9447-16483d800bda-kube-api-access-m8fhq\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-var-lib-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-kubelet-config\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d31e44b4-6f88-42da-b187-34d230a5ded0-hosts-file\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.096996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-run\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.097012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.097025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-slash\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.097046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-ovn\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.097073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-node-log\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.097098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-ovnkube-script-lib\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.097829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.097122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wt6d\" (UniqueName: \"kubernetes.io/projected/d31e44b4-6f88-42da-b187-34d230a5ded0-kube-api-access-7wt6d\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.098560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.098542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.100548 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.100368    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:47:44.100645 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.100611    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 15 05:47:44.101177 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.100985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 15 05:47:44.101177 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.101031    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 15 05:47:44.101177 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.101035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w45dm\""
Jun 15 05:47:44.101177 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.101084    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 15 05:47:44.101415 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.101208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.103377 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.103359    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 15 05:47:44.103470 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.103376    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 15 05:47:44.103470 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.103378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nf5bl\""
Jun 15 05:47:44.103666 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.103644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:44.103782 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.103750    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:44.106096 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.106078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.108053 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.108027    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6r85h\""
Jun 15 05:47:44.108169 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.108083    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 15 05:47:44.108279 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.108253    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 15 05:47:44.108977 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.108443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.110225 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.110206    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 15 05:47:44.110430 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.110296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lkf5v\""
Jun 15 05:47:44.110510 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.110436    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 15 05:47:44.110510 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.110351    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 15 05:47:44.122755 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.122705    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bsmpp"
Jun 15 05:47:44.133506 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.133443    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bsmpp"
Jun 15 05:47:44.189364 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.189337    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jun 15 05:47:44.197891 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.197858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad-agent-certs\") pod \"konnectivity-agent-9c4zl\" (UID: \"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad\") " pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.197891 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.197891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-run\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.198065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.197937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.198065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.197979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:44.198065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.198065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198005    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-run\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.198065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-ovn\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b232addf-4916-4b31-b4b3-fa4157ad304d-cni-binary-copy\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.198065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-systemd\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-etc-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-ovn\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-cni-netd\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-dbus\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198123    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-etc-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysconfig\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-ovnkube-config\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198130    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-systemd\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-cni-netd\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/175a2c40-7bff-4021-a86a-524baa098f52-ovn-node-metrics-cert\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198241    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198214    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-dbus\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31e44b4-6f88-42da-b187-34d230a5ded0-tmp-dir\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198260    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysconfig\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.198331 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-k8s-cni-cncf-io\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad-konnectivity-ca\") pod \"konnectivity-agent-9c4zl\" (UID: \"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad\") " pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-system-cni-dir\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-log-socket\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-cni-multus\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-multus-certs\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198384    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-log-socket\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-device-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysctl-d\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-var-lib-kubelet\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zm7\" (UniqueName: \"kubernetes.io/projected/78c95a18-cd99-4d98-aa17-5334445a567d-kube-api-access-p7zm7\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-env-overrides\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198544    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysctl-d\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198551    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d31e44b4-6f88-42da-b187-34d230a5ded0-tmp-dir\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-var-lib-kubelet\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-system-cni-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198560    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 15 05:47:44.199175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-cnibin\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-kubelet\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-conf-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78c95a18-cd99-4d98-aa17-5334445a567d-etc-tuned\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-run-netns\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-netns\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-cni-binary-copy\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198842    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-run-netns\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysctl-conf\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bda06406-4b48-4943-9447-16483d800bda-host-slash\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fhq\" (UniqueName: \"kubernetes.io/projected/bda06406-4b48-4943-9447-16483d800bda-kube-api-access-m8fhq\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198949    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-ovnkube-config\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198959    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bda06406-4b48-4943-9447-16483d800bda-host-slash\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-var-lib-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.198978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-env-overrides\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199008    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-sysctl-conf\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.199831 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-var-lib-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-kubelet-config\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199135    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-daemon-config\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2lz4\" (UniqueName: \"kubernetes.io/projected/e89ca927-c506-418e-bd11-2a05acdf86e4-kube-api-access-g2lz4\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199195    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-kubelet-config\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkdd\" (UniqueName: \"kubernetes.io/projected/b232addf-4916-4b31-b4b3-fa4157ad304d-kube-api-access-nmkdd\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-registration-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.199282    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11b70477-ceae-4d96-a015-b52d5659b1a6-host\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.199353    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret podName:369df3e2-42a9-4674-a0a9-3ed49caa5dc4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:44.699326576 +0000 UTC m=+3.020243840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret") pod "global-pull-secret-syncer-7qgzl" (UID: "369df3e2-42a9-4674-a0a9-3ed49caa5dc4") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-slash\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-node-log\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-ovnkube-script-lib\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.200532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199427    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-slash\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wt6d\" (UniqueName: \"kubernetes.io/projected/d31e44b4-6f88-42da-b187-34d230a5ded0-kube-api-access-7wt6d\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199456    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-node-log\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-socket-dir-parent\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-kubernetes\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-systemd-units\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-os-release\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-cni-bin\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqpj\" (UniqueName: \"kubernetes.io/projected/6c6082db-426d-49f5-8603-71bab34f199b-kube-api-access-rzqpj\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnw4\" (UniqueName: \"kubernetes.io/projected/11b70477-ceae-4d96-a015-b52d5659b1a6-kube-api-access-wmnw4\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-sys\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199692    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-systemd-units\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199699    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-kubernetes\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-lib-modules\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-host\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-hostroot\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.201034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-sys\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199819    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-host\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-os-release\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199887    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-lib-modules\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11b70477-ceae-4d96-a015-b52d5659b1a6-serviceca\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/175a2c40-7bff-4021-a86a-524baa098f52-ovnkube-script-lib\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-kubelet\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199988    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-kubelet\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.199992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-systemd\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200020    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-systemd\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-cni-bin\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200045    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-host-cni-bin\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl8d\" (UniqueName: \"kubernetes.io/projected/175a2c40-7bff-4021-a86a-524baa098f52-kube-api-access-fdl8d\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-cnibin\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-modprobe-d\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c95a18-cd99-4d98-aa17-5334445a567d-tmp\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.201560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bda06406-4b48-4943-9447-16483d800bda-iptables-alerter-script\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-etc-kubernetes\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200255    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78c95a18-cd99-4d98-aa17-5334445a567d-etc-modprobe-d\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-sys-fs\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hxm\" (UniqueName: \"kubernetes.io/projected/9a3746b9-06bd-418a-b727-fcc71c98d786-kube-api-access-q8hxm\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-socket-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d31e44b4-6f88-42da-b187-34d230a5ded0-hosts-file\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200442    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d31e44b4-6f88-42da-b187-34d230a5ded0-hosts-file\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200463    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/175a2c40-7bff-4021-a86a-524baa098f52-run-openvswitch\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-cni-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.200784    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bda06406-4b48-4943-9447-16483d800bda-iptables-alerter-script\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.201828    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78c95a18-cd99-4d98-aa17-5334445a567d-etc-tuned\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.201854    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/175a2c40-7bff-4021-a86a-524baa098f52-ovn-node-metrics-cert\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.202437 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.202123    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c95a18-cd99-4d98-aa17-5334445a567d-tmp\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.209517 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.209483    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zm7\" (UniqueName: \"kubernetes.io/projected/78c95a18-cd99-4d98-aa17-5334445a567d-kube-api-access-p7zm7\") pod \"tuned-2mvd7\" (UID: \"78c95a18-cd99-4d98-aa17-5334445a567d\") " pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.209785 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.209763    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fhq\" (UniqueName: \"kubernetes.io/projected/bda06406-4b48-4943-9447-16483d800bda-kube-api-access-m8fhq\") pod \"iptables-alerter-kjqsr\" (UID: \"bda06406-4b48-4943-9447-16483d800bda\") " pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.210207 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.210190    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wt6d\" (UniqueName: \"kubernetes.io/projected/d31e44b4-6f88-42da-b187-34d230a5ded0-kube-api-access-7wt6d\") pod \"node-resolver-xcp6n\" (UID: \"d31e44b4-6f88-42da-b187-34d230a5ded0\") " pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.210432 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.210413    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdl8d\" (UniqueName: \"kubernetes.io/projected/175a2c40-7bff-4021-a86a-524baa098f52-kube-api-access-fdl8d\") pod \"ovnkube-node-gnn2f\" (UID: \"175a2c40-7bff-4021-a86a-524baa098f52\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.301302 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-cnibin\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301302 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-kubelet\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301302 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-conf-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301302 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-netns\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-cni-binary-copy\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301351    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-kubelet\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301358    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-conf-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-cnibin\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-daemon-config\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301391    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-netns\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2lz4\" (UniqueName: \"kubernetes.io/projected/e89ca927-c506-418e-bd11-2a05acdf86e4-kube-api-access-g2lz4\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkdd\" (UniqueName: \"kubernetes.io/projected/b232addf-4916-4b31-b4b3-fa4157ad304d-kube-api-access-nmkdd\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-registration-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11b70477-ceae-4d96-a015-b52d5659b1a6-host\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-socket-dir-parent\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-os-release\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.301622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-cni-bin\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqpj\" (UniqueName: \"kubernetes.io/projected/6c6082db-426d-49f5-8603-71bab34f199b-kube-api-access-rzqpj\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnw4\" (UniqueName: \"kubernetes.io/projected/11b70477-ceae-4d96-a015-b52d5659b1a6-kube-api-access-wmnw4\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-hostroot\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-os-release\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11b70477-ceae-4d96-a015-b52d5659b1a6-serviceca\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-cnibin\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301840    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-cni-bin\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-etc-kubernetes\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-sys-fs\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301892    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-registration-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hxm\" (UniqueName: \"kubernetes.io/projected/9a3746b9-06bd-418a-b727-fcc71c98d786-kube-api-access-q8hxm\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301922    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11b70477-ceae-4d96-a015-b52d5659b1a6-host\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-socket-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301961    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-socket-dir-parent\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301953    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-cni-binary-copy\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.302357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-cni-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.301998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad-agent-certs\") pod \"konnectivity-agent-9c4zl\" (UID: \"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad\") " pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302016    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-os-release\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302067    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-etc-kubernetes\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302089    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b232addf-4916-4b31-b4b3-fa4157ad304d-cni-binary-copy\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302118    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-sys-fs\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302168    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-socket-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-k8s-cni-cncf-io\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302221    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-hostroot\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad-konnectivity-ca\") pod \"konnectivity-agent-9c4zl\" (UID: \"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad\") " pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302234    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-k8s-cni-cncf-io\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-system-cni-dir\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302275    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-cnibin\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302296    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-cni-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302333    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-os-release\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302434    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302494    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302527    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c6082db-426d-49f5-8603-71bab34f199b-system-cni-dir\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302542    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c6082db-426d-49f5-8603-71bab34f199b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-cni-multus\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.302563    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-multus-certs\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302619    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b232addf-4916-4b31-b4b3-fa4157ad304d-multus-daemon-config\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-device-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.302665    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:44.802647338 +0000 UTC m=+3.123564608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302679    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-var-lib-cni-multus\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-host-run-multus-certs\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-system-cni-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.303751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302765    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b232addf-4916-4b31-b4b3-fa4157ad304d-cni-binary-copy\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.304350 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302749    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e89ca927-c506-418e-bd11-2a05acdf86e4-device-dir\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.304350 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302821    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b232addf-4916-4b31-b4b3-fa4157ad304d-system-cni-dir\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.304350 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.302878    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad-konnectivity-ca\") pod \"konnectivity-agent-9c4zl\" (UID: \"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad\") " pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.304350 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.303124    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11b70477-ceae-4d96-a015-b52d5659b1a6-serviceca\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.304785 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.304768    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad-agent-certs\") pod \"konnectivity-agent-9c4zl\" (UID: \"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad\") " pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.307926 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.307893    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:44.307926 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.307919    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:44.308122 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.307933    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bnzgh for pod openshift-network-diagnostics/network-check-target-thbgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:44.308122 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.308031    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh podName:915e92a0-de1d-46ad-ba10-bcab485dc2ad nodeName:}" failed. No retries permitted until 2026-06-15 05:47:44.808014648 +0000 UTC m=+3.128931915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bnzgh" (UniqueName: "kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh") pod "network-check-target-thbgq" (UID: "915e92a0-de1d-46ad-ba10-bcab485dc2ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:44.309755 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.309693    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkdd\" (UniqueName: \"kubernetes.io/projected/b232addf-4916-4b31-b4b3-fa4157ad304d-kube-api-access-nmkdd\") pod \"multus-lfbdv\" (UID: \"b232addf-4916-4b31-b4b3-fa4157ad304d\") " pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.309755 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.309700    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2lz4\" (UniqueName: \"kubernetes.io/projected/e89ca927-c506-418e-bd11-2a05acdf86e4-kube-api-access-g2lz4\") pod \"aws-ebs-csi-driver-node-8b6hb\" (UID: \"e89ca927-c506-418e-bd11-2a05acdf86e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.310300 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.310278    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnw4\" (UniqueName: \"kubernetes.io/projected/11b70477-ceae-4d96-a015-b52d5659b1a6-kube-api-access-wmnw4\") pod \"node-ca-2vs2q\" (UID: \"11b70477-ceae-4d96-a015-b52d5659b1a6\") " pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.311523 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.311502    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hxm\" (UniqueName: \"kubernetes.io/projected/9a3746b9-06bd-418a-b727-fcc71c98d786-kube-api-access-q8hxm\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:44.312198 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.312180    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqpj\" (UniqueName: \"kubernetes.io/projected/6c6082db-426d-49f5-8603-71bab34f199b-kube-api-access-rzqpj\") pod \"multus-additional-cni-plugins-822r9\" (UID: \"6c6082db-426d-49f5-8603-71bab34f199b\") " pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.392601 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.392565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcp6n"
Jun 15 05:47:44.400459 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.400439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2mvd7"
Jun 15 05:47:44.409052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.409034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kjqsr"
Jun 15 05:47:44.415745 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.415702    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:47:44.423103 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.423081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2vs2q"
Jun 15 05:47:44.429908 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.429885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lfbdv"
Jun 15 05:47:44.437761 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.437740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-822r9"
Jun 15 05:47:44.444599 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.444579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:47:44.450622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.450595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb"
Jun 15 05:47:44.705332 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.705248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:44.705480 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.705395    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:44.705480 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.705461    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret podName:369df3e2-42a9-4674-a0a9-3ed49caa5dc4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:45.705443471 +0000 UTC m=+4.026360737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret") pod "global-pull-secret-syncer-7qgzl" (UID: "369df3e2-42a9-4674-a0a9-3ed49caa5dc4") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:44.806240 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.806215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:44.806419 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.806331    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:44.806419 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.806405    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:45.806382666 +0000 UTC m=+4.127299933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:44.907032 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:44.907005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:44.907140 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.907115    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:44.907140 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.907128    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:44.907140 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.907139    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bnzgh for pod openshift-network-diagnostics/network-check-target-thbgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:44.907235 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:44.907186    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh podName:915e92a0-de1d-46ad-ba10-bcab485dc2ad nodeName:}" failed. No retries permitted until 2026-06-15 05:47:45.90717403 +0000 UTC m=+4.228091294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnzgh" (UniqueName: "kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh") pod "network-check-target-thbgq" (UID: "915e92a0-de1d-46ad-ba10-bcab485dc2ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:44.914148 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.913990    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89ca927_c506_418e_bd11_2a05acdf86e4.slice/crio-70842339a986cc915c8b0faa6817ed42ffdc9e67904657abb31704bc7f2ebad0 WatchSource:0}: Error finding container 70842339a986cc915c8b0faa6817ed42ffdc9e67904657abb31704bc7f2ebad0: Status 404 returned error can't find the container with id 70842339a986cc915c8b0faa6817ed42ffdc9e67904657abb31704bc7f2ebad0
Jun 15 05:47:44.915439 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.915414    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c95a18_cd99_4d98_aa17_5334445a567d.slice/crio-bb2834c99058efe2bc863e04c6768ea2df3be69774d716cdb65d6c54b42e5970 WatchSource:0}: Error finding container bb2834c99058efe2bc863e04c6768ea2df3be69774d716cdb65d6c54b42e5970: Status 404 returned error can't find the container with id bb2834c99058efe2bc863e04c6768ea2df3be69774d716cdb65d6c54b42e5970
Jun 15 05:47:44.916651 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.916580    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6082db_426d_49f5_8603_71bab34f199b.slice/crio-0df1e39ba5b49d60769e6160b016de507964fa94d234133866a3a17379bdf3d6 WatchSource:0}: Error finding container 0df1e39ba5b49d60769e6160b016de507964fa94d234133866a3a17379bdf3d6: Status 404 returned error can't find the container with id 0df1e39ba5b49d60769e6160b016de507964fa94d234133866a3a17379bdf3d6
Jun 15 05:47:44.919896 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.919867    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6e0b77_c7f7_48a5_907f_9a5dd0ea60ad.slice/crio-4246e76f5af46afbc9ae0bb8f1901396b371d5342fa2f641ca3ae23d4a90f05e WatchSource:0}: Error finding container 4246e76f5af46afbc9ae0bb8f1901396b371d5342fa2f641ca3ae23d4a90f05e: Status 404 returned error can't find the container with id 4246e76f5af46afbc9ae0bb8f1901396b371d5342fa2f641ca3ae23d4a90f05e
Jun 15 05:47:44.920653 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.920620    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb232addf_4916_4b31_b4b3_fa4157ad304d.slice/crio-6a4d42cb0b896c376c2b7698150d8ad76931db6816ec6a0fe24831aad054f996 WatchSource:0}: Error finding container 6a4d42cb0b896c376c2b7698150d8ad76931db6816ec6a0fe24831aad054f996: Status 404 returned error can't find the container with id 6a4d42cb0b896c376c2b7698150d8ad76931db6816ec6a0fe24831aad054f996
Jun 15 05:47:44.921840 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.921814    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175a2c40_7bff_4021_a86a_524baa098f52.slice/crio-6d2eb1cd51216726cdc383e510f9daf25b34e8c7835c583cfce6a3e2cc14d754 WatchSource:0}: Error finding container 6d2eb1cd51216726cdc383e510f9daf25b34e8c7835c583cfce6a3e2cc14d754: Status 404 returned error can't find the container with id 6d2eb1cd51216726cdc383e510f9daf25b34e8c7835c583cfce6a3e2cc14d754
Jun 15 05:47:44.923322 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.923117    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31e44b4_6f88_42da_b187_34d230a5ded0.slice/crio-22ef7ce6b83c2573dfd4b1a6b3b851e109024afa6a85e06e98951569ff1cb2be WatchSource:0}: Error finding container 22ef7ce6b83c2573dfd4b1a6b3b851e109024afa6a85e06e98951569ff1cb2be: Status 404 returned error can't find the container with id 22ef7ce6b83c2573dfd4b1a6b3b851e109024afa6a85e06e98951569ff1cb2be
Jun 15 05:47:44.924528 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.924503    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda06406_4b48_4943_9447_16483d800bda.slice/crio-aba8f19c2941644dcb3fde39d9c152a6ef2d57bcafff4839afa8b43e3c8735b2 WatchSource:0}: Error finding container aba8f19c2941644dcb3fde39d9c152a6ef2d57bcafff4839afa8b43e3c8735b2: Status 404 returned error can't find the container with id aba8f19c2941644dcb3fde39d9c152a6ef2d57bcafff4839afa8b43e3c8735b2
Jun 15 05:47:44.926103 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:47:44.926014    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b70477_ceae_4d96_a015_b52d5659b1a6.slice/crio-8417fc1d9bc478ae5ede8be3570b4bf0298ac504ea3b91ac47da41a51709bf67 WatchSource:0}: Error finding container 8417fc1d9bc478ae5ede8be3570b4bf0298ac504ea3b91ac47da41a51709bf67: Status 404 returned error can't find the container with id 8417fc1d9bc478ae5ede8be3570b4bf0298ac504ea3b91ac47da41a51709bf67
Jun 15 05:47:45.136883 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.136807    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:42:44 +0000 UTC" deadline="2027-12-06 07:37:29.905408433 +0000 UTC"
Jun 15 05:47:45.137151 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.136888    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12937h49m44.768526408s"
Jun 15 05:47:45.154447 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.154422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:45.154606 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.154582    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:45.162615 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.162577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"6d2eb1cd51216726cdc383e510f9daf25b34e8c7835c583cfce6a3e2cc14d754"}
Jun 15 05:47:45.163693 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.163672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerStarted","Data":"0df1e39ba5b49d60769e6160b016de507964fa94d234133866a3a17379bdf3d6"}
Jun 15 05:47:45.164858 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.164836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2mvd7" event={"ID":"78c95a18-cd99-4d98-aa17-5334445a567d","Type":"ContainerStarted","Data":"bb2834c99058efe2bc863e04c6768ea2df3be69774d716cdb65d6c54b42e5970"}
Jun 15 05:47:45.165978 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.165953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb" event={"ID":"e89ca927-c506-418e-bd11-2a05acdf86e4","Type":"ContainerStarted","Data":"70842339a986cc915c8b0faa6817ed42ffdc9e67904657abb31704bc7f2ebad0"}
Jun 15 05:47:45.167883 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.167858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal" event={"ID":"9e3ac6eed3caa2ac33824ee71932a85f","Type":"ContainerStarted","Data":"244db51f09f9aa65cda9ed5c68f8ee8c8645fb5b80c8120841486fa01d0427b9"}
Jun 15 05:47:45.168863 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.168841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2vs2q" event={"ID":"11b70477-ceae-4d96-a015-b52d5659b1a6","Type":"ContainerStarted","Data":"8417fc1d9bc478ae5ede8be3570b4bf0298ac504ea3b91ac47da41a51709bf67"}
Jun 15 05:47:45.169766 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.169714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kjqsr" event={"ID":"bda06406-4b48-4943-9447-16483d800bda","Type":"ContainerStarted","Data":"aba8f19c2941644dcb3fde39d9c152a6ef2d57bcafff4839afa8b43e3c8735b2"}
Jun 15 05:47:45.170643 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.170623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcp6n" event={"ID":"d31e44b4-6f88-42da-b187-34d230a5ded0","Type":"ContainerStarted","Data":"22ef7ce6b83c2573dfd4b1a6b3b851e109024afa6a85e06e98951569ff1cb2be"}
Jun 15 05:47:45.171671 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.171652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lfbdv" event={"ID":"b232addf-4916-4b31-b4b3-fa4157ad304d","Type":"ContainerStarted","Data":"6a4d42cb0b896c376c2b7698150d8ad76931db6816ec6a0fe24831aad054f996"}
Jun 15 05:47:45.172577 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.172553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9c4zl" event={"ID":"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad","Type":"ContainerStarted","Data":"4246e76f5af46afbc9ae0bb8f1901396b371d5342fa2f641ca3ae23d4a90f05e"}
Jun 15 05:47:45.713414 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.713371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:45.713616 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.713525    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:45.713616 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.713590    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret podName:369df3e2-42a9-4674-a0a9-3ed49caa5dc4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:47.71357187 +0000 UTC m=+6.034489156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret") pod "global-pull-secret-syncer-7qgzl" (UID: "369df3e2-42a9-4674-a0a9-3ed49caa5dc4") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:45.813967 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.813927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:45.814155 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.814103    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:45.814213 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.814173    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:47.814151544 +0000 UTC m=+6.135068814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:45.914928 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:45.914887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:45.915132 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.915114    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:45.915197 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.915135    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:45.915197 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.915148    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bnzgh for pod openshift-network-diagnostics/network-check-target-thbgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:45.915299 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:45.915206    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh podName:915e92a0-de1d-46ad-ba10-bcab485dc2ad nodeName:}" failed. No retries permitted until 2026-06-15 05:47:47.915188036 +0000 UTC m=+6.236105303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnzgh" (UniqueName: "kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh") pod "network-check-target-thbgq" (UID: "915e92a0-de1d-46ad-ba10-bcab485dc2ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:46.137641 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:46.137548    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:42:44 +0000 UTC" deadline="2028-02-04 18:05:36.994260192 +0000 UTC"
Jun 15 05:47:46.137641 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:46.137592    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14388h17m50.856672905s"
Jun 15 05:47:46.156301 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:46.156269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:46.156450 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:46.156405    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:46.156834 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:46.156812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:46.156935 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:46.156913    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:46.179455 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:46.179417    2579 generic.go:356] "Generic (PLEG): container finished" podID="385fb380decbfa57906c03876915ce49" containerID="0d8be1741fab8263113fe1d18d75a7cacd34004159f00b4fe0415a094c88a210" exitCode=0
Jun 15 05:47:46.179631 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:46.179501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" event={"ID":"385fb380decbfa57906c03876915ce49","Type":"ContainerDied","Data":"0d8be1741fab8263113fe1d18d75a7cacd34004159f00b4fe0415a094c88a210"}
Jun 15 05:47:46.195026 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:46.194264    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal" podStartSLOduration=3.194232182 podStartE2EDuration="3.194232182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:45.180640585 +0000 UTC m=+3.501557872" watchObservedRunningTime="2026-06-15 05:47:46.194232182 +0000 UTC m=+4.515149472"
Jun 15 05:47:47.154412 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:47.154378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:47.155002 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.154522    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:47.204248 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:47.204209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" event={"ID":"385fb380decbfa57906c03876915ce49","Type":"ContainerStarted","Data":"514074e22317de0e289753aa4f20877b1e430023e50be8d82c6335b116dadcbf"}
Jun 15 05:47:47.228625 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:47.228473    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" podStartSLOduration=4.22845443 podStartE2EDuration="4.22845443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:47.22809964 +0000 UTC m=+5.549016927" watchObservedRunningTime="2026-06-15 05:47:47.22845443 +0000 UTC m=+5.549371719"
Jun 15 05:47:47.733019 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:47.732973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:47.733232 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.733213    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:47.733759 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.733296    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret podName:369df3e2-42a9-4674-a0a9-3ed49caa5dc4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:51.733273656 +0000 UTC m=+10.054190924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret") pod "global-pull-secret-syncer-7qgzl" (UID: "369df3e2-42a9-4674-a0a9-3ed49caa5dc4") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:47.833533 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:47.833488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:47.833709 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.833688    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:47.833810 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.833773    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:51.833752978 +0000 UTC m=+10.154670245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:47.934804 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:47.934752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:47.935018 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.934999    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:47.935094 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.935023    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:47.935094 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.935036    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bnzgh for pod openshift-network-diagnostics/network-check-target-thbgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:47.935198 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:47.935094    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh podName:915e92a0-de1d-46ad-ba10-bcab485dc2ad nodeName:}" failed. No retries permitted until 2026-06-15 05:47:51.935076842 +0000 UTC m=+10.255994113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnzgh" (UniqueName: "kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh") pod "network-check-target-thbgq" (UID: "915e92a0-de1d-46ad-ba10-bcab485dc2ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:48.154857 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:48.154773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:48.154857 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:48.154803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:48.155338 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:48.154922    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:48.155338 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:48.155037    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:49.154453 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:49.154419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:49.154661 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:49.154569    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:50.156238 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:50.156180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:50.156238 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:50.156235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:50.156752 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:50.156319    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:50.156752 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:50.156669    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:51.154334 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:51.154295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:51.154520 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.154451    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:51.769328 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:51.769292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:51.769804 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.769491    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:51.769804 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.769541    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret podName:369df3e2-42a9-4674-a0a9-3ed49caa5dc4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:59.769527232 +0000 UTC m=+18.090444495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret") pod "global-pull-secret-syncer-7qgzl" (UID: "369df3e2-42a9-4674-a0a9-3ed49caa5dc4") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:51.870823 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:51.870686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:51.871011 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.870871    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:51.871011 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.870948    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:59.870926188 +0000 UTC m=+18.191843454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:51.972000 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:51.971953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:51.972159 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.972149    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:51.972197 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.972164    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:51.972197 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.972175    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bnzgh for pod openshift-network-diagnostics/network-check-target-thbgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:51.972271 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:51.972224    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh podName:915e92a0-de1d-46ad-ba10-bcab485dc2ad nodeName:}" failed. No retries permitted until 2026-06-15 05:47:59.972210657 +0000 UTC m=+18.293127921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnzgh" (UniqueName: "kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh") pod "network-check-target-thbgq" (UID: "915e92a0-de1d-46ad-ba10-bcab485dc2ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:52.157089 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:52.157006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:52.157269 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:52.157248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:52.157344 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:52.157278    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:52.157422 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:52.157395    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:53.154845 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:53.154749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:53.155300 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:53.154909    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:54.154193 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:54.154154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:54.154409 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:54.154294    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:54.154501 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:54.154480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:54.154595 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:54.154575    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:55.154198 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:55.154158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:55.154635 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:55.154277    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:56.154417 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:56.154383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:56.154930 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:56.154427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:56.154930 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:56.154523    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:56.154930 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:56.154624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:57.154558 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:57.154521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:57.155037 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:57.154711    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:58.158068 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:58.158023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:47:58.158611 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:58.158021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:58.158611 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:58.158156    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:47:58.158611 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:58.158240    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:47:59.154668 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:59.154630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:59.154886 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:59.154821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:47:59.832183 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:59.832145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:47:59.832636 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:59.832294    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:59.832636 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:59.832369    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret podName:369df3e2-42a9-4674-a0a9-3ed49caa5dc4 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:15.83234986 +0000 UTC m=+34.153267137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret") pod "global-pull-secret-syncer-7qgzl" (UID: "369df3e2-42a9-4674-a0a9-3ed49caa5dc4") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:59.932550 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:47:59.932511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:47:59.932719 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:59.932700    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:59.932821 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:47:59.932794    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:15.932773635 +0000 UTC m=+34.253690899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:00.033362 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:00.033323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:00.033516 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:00.033495    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:00.033564 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:00.033517    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:00.033564 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:00.033528    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bnzgh for pod openshift-network-diagnostics/network-check-target-thbgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:00.033658 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:00.033587    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh podName:915e92a0-de1d-46ad-ba10-bcab485dc2ad nodeName:}" failed. No retries permitted until 2026-06-15 05:48:16.033568481 +0000 UTC m=+34.354485747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnzgh" (UniqueName: "kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh") pod "network-check-target-thbgq" (UID: "915e92a0-de1d-46ad-ba10-bcab485dc2ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:00.156837 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:00.156758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:00.156837 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:00.156799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:00.157042 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:00.156888    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:00.157042 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:00.157022    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:01.153922 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:01.153888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:01.154325 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:01.154006    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:02.155135 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:02.155102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:02.155609 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:02.155201    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:02.155609 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:02.155267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:02.155609 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:02.155378    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:03.154738 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:03.154685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:03.154897 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:03.154819    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:04.154426 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.154134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:04.155122 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.154143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:04.155122 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:04.154524    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:04.155122 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:04.154599    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:04.243246 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.243164    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c6082db-426d-49f5-8603-71bab34f199b" containerID="e674fc4d8c72cf15064ba9b5adc138a70f345959e397a9111b5bcacf9b8a4ea0" exitCode=0
Jun 15 05:48:04.243815 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.243468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerDied","Data":"e674fc4d8c72cf15064ba9b5adc138a70f345959e397a9111b5bcacf9b8a4ea0"}
Jun 15 05:48:04.245245 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.245221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2mvd7" event={"ID":"78c95a18-cd99-4d98-aa17-5334445a567d","Type":"ContainerStarted","Data":"58adbecdfe73d822379762f6fdb080cbbd963f5b3ec2face2bed99707f5fd28e"}
Jun 15 05:48:04.247356 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.247331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb" event={"ID":"e89ca927-c506-418e-bd11-2a05acdf86e4","Type":"ContainerStarted","Data":"a96290a3a9a55352360d89428f245b4aec6f4ae39cc482e14f62ef702289d963"}
Jun 15 05:48:04.249190 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.249162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2vs2q" event={"ID":"11b70477-ceae-4d96-a015-b52d5659b1a6","Type":"ContainerStarted","Data":"8b48af99728e0e51dbab17300590c56c494bdc99b712b06abb2efc5e99724a1f"}
Jun 15 05:48:04.251017 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.250971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcp6n" event={"ID":"d31e44b4-6f88-42da-b187-34d230a5ded0","Type":"ContainerStarted","Data":"d776bd20e41fe07cec2f506d68018a85c6cd4fd62b6cbcdd911afdc8dac9c8aa"}
Jun 15 05:48:04.252698 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.252634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lfbdv" event={"ID":"b232addf-4916-4b31-b4b3-fa4157ad304d","Type":"ContainerStarted","Data":"9bb7aa0f963d5cb7fc795e375fb5f2eb88708bfcc83ee613928102989c141d6d"}
Jun 15 05:48:04.254242 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.254197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9c4zl" event={"ID":"be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad","Type":"ContainerStarted","Data":"a2d00aec77c6e4f835cfd0a4cf32de6f62d2e9cf5e70d4c08c412f5ef865e6c8"}
Jun 15 05:48:04.260345 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.260323    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:48:04.260815 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.260789    2579 generic.go:356] "Generic (PLEG): container finished" podID="175a2c40-7bff-4021-a86a-524baa098f52" containerID="9c52fec40d559dcdb8008e49d73c00bd0148117fe1475430c29a57371ecd5710" exitCode=1
Jun 15 05:48:04.260925 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.260825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"06454af0c90ce5e16d8cd57608f37a8242c1accaf92a0408fc6564174ceab0ec"}
Jun 15 05:48:04.260925 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.260878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"e915fc735d9499649ba94b1dbadaff6625c4ff6ae32941828422a543f2610250"}
Jun 15 05:48:04.260925 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.260913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"178fece99eb70eecfce4ff752f1bf5cfbc6747572fa91f36610b12a9e76dbb41"}
Jun 15 05:48:04.261084 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.260963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"7244b47dca635b4343962fc042ac7f0bc70b06d42ddfbb8fc646e71c5d0e1066"}
Jun 15 05:48:04.261084 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.261012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerDied","Data":"9c52fec40d559dcdb8008e49d73c00bd0148117fe1475430c29a57371ecd5710"}
Jun 15 05:48:04.261084 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.261060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"186fa34e725e98dd0f83e2c2b81d80a1859c8e6b2dac5a6e7346ff15e4798e72"}
Jun 15 05:48:04.277300 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.277246    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2vs2q" podStartSLOduration=3.7204783409999997 podStartE2EDuration="22.277226659s" totalImagesPullingTime="18.556748318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:04.27661115 +0000 UTC m=+22.597528437" watchObservedRunningTime="2026-06-15 05:48:04.277226659 +0000 UTC m=+22.598143947"
Jun 15 05:48:04.289157 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.289110    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xcp6n" podStartSLOduration=3.729338127 podStartE2EDuration="22.289095686s" totalImagesPullingTime="18.559757559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:04.288818603 +0000 UTC m=+22.609735891" watchObservedRunningTime="2026-06-15 05:48:04.289095686 +0000 UTC m=+22.610012971"
Jun 15 05:48:04.315283 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.315230    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lfbdv" podStartSLOduration=3.708799932 podStartE2EDuration="22.315214509s" totalImagesPullingTime="18.606414577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:04.314996061 +0000 UTC m=+22.635913347" watchObservedRunningTime="2026-06-15 05:48:04.315214509 +0000 UTC m=+22.636131795"
Jun 15 05:48:04.329916 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:04.329866    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9c4zl" podStartSLOduration=7.88554324 podStartE2EDuration="22.329849277s" totalImagesPullingTime="14.444306037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:04.329777136 +0000 UTC m=+22.650694422" watchObservedRunningTime="2026-06-15 05:48:04.329849277 +0000 UTC m=+22.650766563"
Jun 15 05:48:05.115058 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.115026    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 15 05:48:05.154903 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.154792    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-15T05:48:05.115050776Z","UUID":"58450ede-cee1-45a1-afa9-787ac89b0dfe","Handler":null,"Name":"","Endpoint":""}
Jun 15 05:48:05.154903 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.154881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:05.155425 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:05.155013    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:05.156548 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.156525    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 15 05:48:05.156680 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.156596    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 15 05:48:05.265656 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.265562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb" event={"ID":"e89ca927-c506-418e-bd11-2a05acdf86e4","Type":"ContainerStarted","Data":"ef58789bae5a10af6faab23b75efa70069f32cdf8ca4c033b0baa8d1827a307f"}
Jun 15 05:48:05.267249 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.267218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kjqsr" event={"ID":"bda06406-4b48-4943-9447-16483d800bda","Type":"ContainerStarted","Data":"25c5ac29229e4f0d792c5b65e817dd108f5091ad55380c4cc2c6fe020cf9e6a3"}
Jun 15 05:48:05.281528 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.281484    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kjqsr" podStartSLOduration=4.723763688 podStartE2EDuration="23.28146567s" totalImagesPullingTime="18.557701982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:05.281298372 +0000 UTC m=+23.602215659" watchObservedRunningTime="2026-06-15 05:48:05.28146567 +0000 UTC m=+23.602382955"
Jun 15 05:48:05.282150 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:05.282112    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2mvd7" podStartSLOduration=4.712649382 podStartE2EDuration="23.282100873s" totalImagesPullingTime="18.569451491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:04.345405105 +0000 UTC m=+22.666322392" watchObservedRunningTime="2026-06-15 05:48:05.282100873 +0000 UTC m=+23.603018189"
Jun 15 05:48:06.154870 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:06.154836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:06.155038 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:06.154977    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:06.155389 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:06.155245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:06.155389 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:06.155358    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:06.272101 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:06.271916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb" event={"ID":"e89ca927-c506-418e-bd11-2a05acdf86e4","Type":"ContainerStarted","Data":"3a80dd52a438a7051c0e5f376ae795a1adf5435ba2d91e48fc4b4e8261d49222"}
Jun 15 05:48:06.288654 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:06.288609    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6hb" podStartSLOduration=3.518326803 podStartE2EDuration="24.288593822s" totalImagesPullingTime="20.770267019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:06.288464197 +0000 UTC m=+24.609381484" watchObservedRunningTime="2026-06-15 05:48:06.288593822 +0000 UTC m=+24.609511107"
Jun 15 05:48:06.501242 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:06.501151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:48:06.501865 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:06.501838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:48:07.154077 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:07.154044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:07.154278 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:07.154179    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:07.277995 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:07.277969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:48:07.278516 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:07.278413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"3bf7abd1575fea25bd20b7f8629819f258dbeff50783dcb40b8ececc05354327"}
Jun 15 05:48:07.278887 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:07.278851    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:48:07.279132 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:07.279114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9c4zl"
Jun 15 05:48:08.153989 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:08.153953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:08.154173 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:08.153970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:08.154173 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:08.154095    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:08.154255 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:08.154169    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:09.154675 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.154498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:09.155329 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:09.154776    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:09.286857 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.286829    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:48:09.287195 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.287171    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"94e16df9430610e7ecde8f75514a1a90c8f8b4dc373a9b312e75b0c3d2c75478"}
Jun 15 05:48:09.287448 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.287413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:48:09.287560 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.287486    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:48:09.287668 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.287648    2579 scope.go:117] "RemoveContainer" containerID="9c52fec40d559dcdb8008e49d73c00bd0148117fe1475430c29a57371ecd5710"
Jun 15 05:48:09.289169 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.289142    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c6082db-426d-49f5-8603-71bab34f199b" containerID="4a4d23416655012d3d403f63365be7aa3ae3e0a5db243ea92ae603bc0df2215c" exitCode=0
Jun 15 05:48:09.289264 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.289221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerDied","Data":"4a4d23416655012d3d403f63365be7aa3ae3e0a5db243ea92ae603bc0df2215c"}
Jun 15 05:48:09.305205 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.305184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:48:09.305341 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:09.305328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:48:10.154966 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.154936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:10.155354 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.154936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:10.155354 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:10.155045    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:10.155354 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:10.155157    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:10.294919 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.294846    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:48:10.295259 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.295233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" event={"ID":"175a2c40-7bff-4021-a86a-524baa098f52","Type":"ContainerStarted","Data":"97a099e6f127234eea50e15994427600596ccce3eb3ccc792862d5306334a932"}
Jun 15 05:48:10.295324 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.295303    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jun 15 05:48:10.297273 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.297246    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c6082db-426d-49f5-8603-71bab34f199b" containerID="912c70fb606fc0df1289cbfeeb8a1f8a36f1891df2993ba15fc7056e94899ad5" exitCode=0
Jun 15 05:48:10.297357 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.297284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerDied","Data":"912c70fb606fc0df1289cbfeeb8a1f8a36f1891df2993ba15fc7056e94899ad5"}
Jun 15 05:48:10.322491 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.322437    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f" podStartSLOduration=9.727027613 podStartE2EDuration="28.322421016s" totalImagesPullingTime="18.595393403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:10.321110754 +0000 UTC m=+28.642028055" watchObservedRunningTime="2026-06-15 05:48:10.322421016 +0000 UTC m=+28.643338302"
Jun 15 05:48:10.632470 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.632435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7qgzl"]
Jun 15 05:48:10.632622 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.632566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:10.632698 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:10.632675    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:10.635748 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.635693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-thbgq"]
Jun 15 05:48:10.635877 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.635808    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:10.635948 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:10.635911    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:10.636442 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.636421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-smsbk"]
Jun 15 05:48:10.636548 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:10.636535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:10.636678 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:10.636656    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:11.301995 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:11.301911    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c6082db-426d-49f5-8603-71bab34f199b" containerID="067868aff5339bff61396cfd8fa65ddf23fe973aa699a7dab1b091043b71749c" exitCode=0
Jun 15 05:48:11.301995 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:11.301983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerDied","Data":"067868aff5339bff61396cfd8fa65ddf23fe973aa699a7dab1b091043b71749c"}
Jun 15 05:48:11.302382 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:11.302076    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jun 15 05:48:11.697862 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:11.697827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:48:12.155008 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:12.154977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:12.155325 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:12.155060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:12.155325 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:12.155101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:12.155325 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:12.155117    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:12.155325 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:12.155175    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:12.155325 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:12.155250    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:14.154175 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:14.153975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:14.154805 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:14.153976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:14.154805 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:14.154272    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:14.154805 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:14.154014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:14.154805 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:14.154356    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:14.154805 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:14.154448    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:15.844447 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:15.844359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:15.845072 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:15.844526    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:15.845072 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:15.844603    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret podName:369df3e2-42a9-4674-a0a9-3ed49caa5dc4 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:47.844583128 +0000 UTC m=+66.165500392 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret") pod "global-pull-secret-syncer-7qgzl" (UID: "369df3e2-42a9-4674-a0a9-3ed49caa5dc4") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:15.945583 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:15.945542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:15.945793 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:15.945690    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:15.945793 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:15.945783    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:47.945768059 +0000 UTC m=+66.266685329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:16.046051 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.046009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:16.046275 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.046158    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:16.046275 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.046180    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:16.046275 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.046191    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bnzgh for pod openshift-network-diagnostics/network-check-target-thbgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:16.046275 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.046258    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh podName:915e92a0-de1d-46ad-ba10-bcab485dc2ad nodeName:}" failed. No retries permitted until 2026-06-15 05:48:48.046240764 +0000 UTC m=+66.367158045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnzgh" (UniqueName: "kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh") pod "network-check-target-thbgq" (UID: "915e92a0-de1d-46ad-ba10-bcab485dc2ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:16.154165 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.154136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:16.154165 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.154155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:16.154165 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.154173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:16.154488 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.154253    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7qgzl" podUID="369df3e2-42a9-4674-a0a9-3ed49caa5dc4"
Jun 15 05:48:16.154488 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.154411    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:48:16.154488 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.154476    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-thbgq" podUID="915e92a0-de1d-46ad-ba10-bcab485dc2ad"
Jun 15 05:48:16.315968 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.315937    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c6082db-426d-49f5-8603-71bab34f199b" containerID="539bc438cfe51cbe5551c1ab287b3643ebd69c9e96e5cc12b32910bbfd19c15f" exitCode=0
Jun 15 05:48:16.316123 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.315982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerDied","Data":"539bc438cfe51cbe5551c1ab287b3643ebd69c9e96e5cc12b32910bbfd19c15f"}
Jun 15 05:48:16.458823 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.458798    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeReady"
Jun 15 05:48:16.458971 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.458960    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jun 15 05:48:16.491218 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.491185    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"]
Jun 15 05:48:16.509121 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.509090    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"]
Jun 15 05:48:16.509305 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.509223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:16.511409 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.511387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jun 15 05:48:16.511678 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.511660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jun 15 05:48:16.511817 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.511738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8k6cb\""
Jun 15 05:48:16.521480 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.521459    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-757f768787-b5jx5"]
Jun 15 05:48:16.521608 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.521594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.526137 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.526056    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jun 15 05:48:16.526511 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.526488    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jun 15 05:48:16.526627 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.526609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jun 15 05:48:16.526627 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.526607    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jun 15 05:48:16.526976 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.526950    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zjgkg\""
Jun 15 05:48:16.533796 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.533637    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"]
Jun 15 05:48:16.533955 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.533801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.536072 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.536053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jun 15 05:48:16.536358 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.536340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j676s\""
Jun 15 05:48:16.536642 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.536621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jun 15 05:48:16.536761 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.536654    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jun 15 05:48:16.540429 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.540409    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jun 15 05:48:16.545738 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.545706    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"]
Jun 15 05:48:16.545835 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.545812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.548847 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.548828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jun 15 05:48:16.548847 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.548829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jun 15 05:48:16.548990 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.548880    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jun 15 05:48:16.548990 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.548879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jun 15 05:48:16.559451 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.559420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:16.559567 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.559552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"]
Jun 15 05:48:16.559614 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.559581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"]
Jun 15 05:48:16.559614 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.559593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"]
Jun 15 05:48:16.559614 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.559606    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"]
Jun 15 05:48:16.559715 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.559622    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zpd25"]
Jun 15 05:48:16.559715 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.559666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.561812 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.561791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jun 15 05:48:16.571748 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.571714    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x8n46"]
Jun 15 05:48:16.571846 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.571811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.573901 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.573881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 15 05:48:16.574004 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.573911    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d82dw\""
Jun 15 05:48:16.574004 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.573931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 15 05:48:16.584000 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.583978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757f768787-b5jx5"]
Jun 15 05:48:16.584127 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.584012    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zpd25"]
Jun 15 05:48:16.584127 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.584036    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x8n46"]
Jun 15 05:48:16.584127 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.584109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:16.586202 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.586183    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 15 05:48:16.586671 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.586506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 15 05:48:16.586671 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.586510    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 15 05:48:16.586671 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.586512    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rqwfl\""
Jun 15 05:48:16.644417 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.644386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:16.650458 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1345c494-9ebb-4787-b869-3fd018e003b7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.650589 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.650589 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1345c494-9ebb-4787-b869-3fd018e003b7-tmp\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.650589 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.650589 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrjf\" (UniqueName: \"kubernetes.io/projected/b9c0bd7a-8b7d-401b-a301-d4843626a780-kube-api-access-8wrjf\") pod \"managed-serviceaccount-addon-agent-7d75854d66-2b8wk\" (UID: \"b9c0bd7a-8b7d-401b-a301-d4843626a780\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.650778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-trusted-ca\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.650778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1947ce46-e2e0-45af-ae40-738b891e5894-cert\") pod \"ingress-canary-x8n46\" (UID: \"1947ce46-e2e0-45af-ae40-738b891e5894\") " pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:16.650778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-installation-pull-secrets\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.650778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-ca\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.650778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8hld\" (UniqueName: \"kubernetes.io/projected/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-kube-api-access-w8hld\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.650778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/415c4a6f-c939-42f7-9c11-c1730085b403-nginx-conf\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:16.650778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-config-volume\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-registry-certificates\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7dd\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-kube-api-access-vt7dd\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdkw\" (UniqueName: \"kubernetes.io/projected/1947ce46-e2e0-45af-ae40-738b891e5894-kube-api-access-vkdkw\") pod \"ingress-canary-x8n46\" (UID: \"1947ce46-e2e0-45af-ae40-738b891e5894\") " pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfed647-4879-412d-82ac-c256224b3d0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-tmp-dir\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-hub\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwkm\" (UniqueName: \"kubernetes.io/projected/1dfed647-4879-412d-82ac-c256224b3d0c-kube-api-access-gfwkm\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.650997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-bound-sa-token\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.651018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9c0bd7a-8b7d-401b-a301-d4843626a780-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d75854d66-2b8wk\" (UID: \"b9c0bd7a-8b7d-401b-a301-d4843626a780\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.651034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.651052 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.651051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.651398 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.651077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-image-registry-private-configuration\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.651398 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.651094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/1345c494-9ebb-4787-b869-3fd018e003b7-kube-api-access-6l726\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.651398 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.651114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/296f149e-037e-4f23-91d4-d248e11e20ee-ca-trust-extracted\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.752495 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdkw\" (UniqueName: \"kubernetes.io/projected/1947ce46-e2e0-45af-ae40-738b891e5894-kube-api-access-vkdkw\") pod \"ingress-canary-x8n46\" (UID: \"1947ce46-e2e0-45af-ae40-738b891e5894\") " pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:16.752495 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfed647-4879-412d-82ac-c256224b3d0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.752495 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-tmp-dir\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-hub\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwkm\" (UniqueName: \"kubernetes.io/projected/1dfed647-4879-412d-82ac-c256224b3d0c-kube-api-access-gfwkm\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-bound-sa-token\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9c0bd7a-8b7d-401b-a301-d4843626a780-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d75854d66-2b8wk\" (UID: \"b9c0bd7a-8b7d-401b-a301-d4843626a780\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.752674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.752720 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.752687    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:48:16.753144 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.752774    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:17.252750725 +0000 UTC m=+35.573667992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:48:16.753144 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.753121    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-tmp-dir\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.753380 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.753354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-image-registry-private-configuration\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.753475 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.753408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/1345c494-9ebb-4787-b869-3fd018e003b7-kube-api-access-6l726\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.753475 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.753440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/296f149e-037e-4f23-91d4-d248e11e20ee-ca-trust-extracted\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.753573 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.753471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1345c494-9ebb-4787-b869-3fd018e003b7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.753721 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.753700    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfed647-4879-412d-82ac-c256224b3d0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.753952 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.753934    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:48:16.754048 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.753955    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:48:16.754048 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.754020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:48:17.25400329 +0000 UTC m=+35.574920554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:48:16.754881 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.754793    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/296f149e-037e-4f23-91d4-d248e11e20ee-ca-trust-extracted\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.755041 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.754916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.755041 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.754954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1345c494-9ebb-4787-b869-3fd018e003b7-tmp\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.755041 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.755208 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrjf\" (UniqueName: \"kubernetes.io/projected/b9c0bd7a-8b7d-401b-a301-d4843626a780-kube-api-access-8wrjf\") pod \"managed-serviceaccount-addon-agent-7d75854d66-2b8wk\" (UID: \"b9c0bd7a-8b7d-401b-a301-d4843626a780\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.755208 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-trusted-ca\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.755208 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1947ce46-e2e0-45af-ae40-738b891e5894-cert\") pod \"ingress-canary-x8n46\" (UID: \"1947ce46-e2e0-45af-ae40-738b891e5894\") " pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:16.755208 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-installation-pull-secrets\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.755208 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-ca\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.755444 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8hld\" (UniqueName: \"kubernetes.io/projected/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-kube-api-access-w8hld\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.755444 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/415c4a6f-c939-42f7-9c11-c1730085b403-nginx-conf\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:16.755444 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-config-volume\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.755444 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-registry-certificates\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.755444 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.755380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7dd\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-kube-api-access-vt7dd\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.756507 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.756471    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/415c4a6f-c939-42f7-9c11-c1730085b403-nginx-conf\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:16.756610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.756575    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:16.756668 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:16.756647    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:48:17.256629388 +0000 UTC m=+35.577546654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:16.757224 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.757198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1345c494-9ebb-4787-b869-3fd018e003b7-tmp\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.758181 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.758154    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-config-volume\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.758323 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.758296    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9c0bd7a-8b7d-401b-a301-d4843626a780-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d75854d66-2b8wk\" (UID: \"b9c0bd7a-8b7d-401b-a301-d4843626a780\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.758470 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.758349    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-registry-certificates\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.758621 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.758599    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-trusted-ca\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.759055 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.759034    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-image-registry-private-configuration\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.759502 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.759482    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-ca\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.759641 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.759612    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1345c494-9ebb-4787-b869-3fd018e003b7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.760017 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.759990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-hub\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.760897 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.760874    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.761455 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.761289    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-installation-pull-secrets\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.762772 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.762707    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1947ce46-e2e0-45af-ae40-738b891e5894-cert\") pod \"ingress-canary-x8n46\" (UID: \"1947ce46-e2e0-45af-ae40-738b891e5894\") " pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:16.763829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.763805    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwkm\" (UniqueName: \"kubernetes.io/projected/1dfed647-4879-412d-82ac-c256224b3d0c-kube-api-access-gfwkm\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.764198 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.764153    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/1345c494-9ebb-4787-b869-3fd018e003b7-kube-api-access-6l726\") pod \"klusterlet-addon-workmgr-7d5cbb759-qcmzx\" (UID: \"1345c494-9ebb-4787-b869-3fd018e003b7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.765976 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.765341    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7dd\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-kube-api-access-vt7dd\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.765976 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.765902    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dfed647-4879-412d-82ac-c256224b3d0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d5b856bb6-cp4lt\" (UID: \"1dfed647-4879-412d-82ac-c256224b3d0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.765976 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.765926    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrjf\" (UniqueName: \"kubernetes.io/projected/b9c0bd7a-8b7d-401b-a301-d4843626a780-kube-api-access-8wrjf\") pod \"managed-serviceaccount-addon-agent-7d75854d66-2b8wk\" (UID: \"b9c0bd7a-8b7d-401b-a301-d4843626a780\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.766190 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.766052    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-bound-sa-token\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:16.766300 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.766275    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdkw\" (UniqueName: \"kubernetes.io/projected/1947ce46-e2e0-45af-ae40-738b891e5894-kube-api-access-vkdkw\") pod \"ingress-canary-x8n46\" (UID: \"1947ce46-e2e0-45af-ae40-738b891e5894\") " pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:16.766360 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.766279    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8hld\" (UniqueName: \"kubernetes.io/projected/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-kube-api-access-w8hld\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:16.882685 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.882652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:48:16.911039 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.911000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:48:16.917515 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.917488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:16.927793 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:16.927763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x8n46"
Jun 15 05:48:17.113276 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.113024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"]
Jun 15 05:48:17.119746 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:48:17.119697    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dfed647_4879_412d_82ac_c256224b3d0c.slice/crio-ed0b4145412f037f0c0b496850133bc8f07b5ba2aa033a3f8c5b85c818397ee1 WatchSource:0}: Error finding container ed0b4145412f037f0c0b496850133bc8f07b5ba2aa033a3f8c5b85c818397ee1: Status 404 returned error can't find the container with id ed0b4145412f037f0c0b496850133bc8f07b5ba2aa033a3f8c5b85c818397ee1
Jun 15 05:48:17.127513 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.127478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"]
Jun 15 05:48:17.131453 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:48:17.131429    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9c0bd7a_8b7d_401b_a301_d4843626a780.slice/crio-fc0e214da232ea8ae7229c7ec250b2beb19669a9a0347240a046bc1d7d9cf8af WatchSource:0}: Error finding container fc0e214da232ea8ae7229c7ec250b2beb19669a9a0347240a046bc1d7d9cf8af: Status 404 returned error can't find the container with id fc0e214da232ea8ae7229c7ec250b2beb19669a9a0347240a046bc1d7d9cf8af
Jun 15 05:48:17.134050 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.134029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"]
Jun 15 05:48:17.137100 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:48:17.137077    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1345c494_9ebb_4787_b869_3fd018e003b7.slice/crio-d731acb93aebf4b69a40f7d3dccfc1882685fce64153f14369b3d9abee5155d5 WatchSource:0}: Error finding container d731acb93aebf4b69a40f7d3dccfc1882685fce64153f14369b3d9abee5155d5: Status 404 returned error can't find the container with id d731acb93aebf4b69a40f7d3dccfc1882685fce64153f14369b3d9abee5155d5
Jun 15 05:48:17.159574 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.159546    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x8n46"]
Jun 15 05:48:17.162182 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:48:17.162157    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1947ce46_e2e0_45af_ae40_738b891e5894.slice/crio-bffb51cd1fb0c25748923fdae5c099af0b7cfd7ad7107b8c88082317c6cbf8e3 WatchSource:0}: Error finding container bffb51cd1fb0c25748923fdae5c099af0b7cfd7ad7107b8c88082317c6cbf8e3: Status 404 returned error can't find the container with id bffb51cd1fb0c25748923fdae5c099af0b7cfd7ad7107b8c88082317c6cbf8e3
Jun 15 05:48:17.261481 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.261395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:17.261481 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.261433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:17.261481 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.261470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:17.261691 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:17.261571    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:17.261691 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:17.261574    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:48:17.261691 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:17.261618    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:48:18.261605143 +0000 UTC m=+36.582522407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:17.261691 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:17.261577    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:48:17.261691 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:17.261632    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:48:17.261691 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:17.261632    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:18.261625457 +0000 UTC m=+36.582542721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:48:17.261691 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:17.261673    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:48:18.261655657 +0000 UTC m=+36.582572925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:48:17.322207 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.322170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x8n46" event={"ID":"1947ce46-e2e0-45af-ae40-738b891e5894","Type":"ContainerStarted","Data":"bffb51cd1fb0c25748923fdae5c099af0b7cfd7ad7107b8c88082317c6cbf8e3"}
Jun 15 05:48:17.323425 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.323392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" event={"ID":"1345c494-9ebb-4787-b869-3fd018e003b7","Type":"ContainerStarted","Data":"d731acb93aebf4b69a40f7d3dccfc1882685fce64153f14369b3d9abee5155d5"}
Jun 15 05:48:17.324465 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.324440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" event={"ID":"1dfed647-4879-412d-82ac-c256224b3d0c","Type":"ContainerStarted","Data":"ed0b4145412f037f0c0b496850133bc8f07b5ba2aa033a3f8c5b85c818397ee1"}
Jun 15 05:48:17.327428 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.327400    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c6082db-426d-49f5-8603-71bab34f199b" containerID="25770a5b3f79f1ea246fb734f96051515390db4140290dbb74ed20bc37128c9a" exitCode=0
Jun 15 05:48:17.327543 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.327488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerDied","Data":"25770a5b3f79f1ea246fb734f96051515390db4140290dbb74ed20bc37128c9a"}
Jun 15 05:48:17.328536 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:17.328512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" event={"ID":"b9c0bd7a-8b7d-401b-a301-d4843626a780","Type":"ContainerStarted","Data":"fc0e214da232ea8ae7229c7ec250b2beb19669a9a0347240a046bc1d7d9cf8af"}
Jun 15 05:48:18.155428 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.154849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:18.155428 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.154873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:18.155428 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.155318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:18.160750 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.158434    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:48:18.160750 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.159422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:48:18.160750 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.159674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qfkm9\""
Jun 15 05:48:18.160750 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.159986    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:48:18.160750 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.160173    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:48:18.160750 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.160356    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdlts\""
Jun 15 05:48:18.272806 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.272206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.273116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.273199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:18.273322    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:18.273366    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:48:20.273351656 +0000 UTC m=+38.594268919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:18.273739    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:18.273852    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:18.273863    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:48:18.273946 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:18.273919    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:48:20.273893005 +0000 UTC m=+38.594810288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:48:18.274491 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:18.274474    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:20.274454259 +0000 UTC m=+38.595371537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:48:18.297791 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.297660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:18.340616 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.340580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-822r9" event={"ID":"6c6082db-426d-49f5-8603-71bab34f199b","Type":"ContainerStarted","Data":"560ae3119fee291426a518095bcbbbdad7d7952053ec4803b67139861b5b2f5c"}
Jun 15 05:48:18.352396 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.352310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:18.365453 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.365394    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-822r9" podStartSLOduration=7.961460819 podStartE2EDuration="36.365373046s" totalImagesPullingTime="28.403912227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:18.364304863 +0000 UTC m=+36.685222151" watchObservedRunningTime="2026-06-15 05:48:18.365373046 +0000 UTC m=+36.686290335"
Jun 15 05:48:18.421460 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.421374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:18.481531 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.481487    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:18.642072 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.642039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:18.712455 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:18.711350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:19.053512 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:19.053239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:19.107416 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:19.107381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:19.446419 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:19.446349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:19.500619 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:19.500541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:20.244960 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:20.244868    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:20.298602 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:20.298554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:20.298814 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:20.298676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:20.298814 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:20.298713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:20.298947 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:20.298879    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:48:20.298947 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:20.298892    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:48:20.298947 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:20.298941    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:48:24.298926906 +0000 UTC m=+42.619844171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:48:20.299385 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:20.299218    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:48:20.299385 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:20.299308    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:24.299279189 +0000 UTC m=+42.620196462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:48:20.299385 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:20.299230    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:20.299385 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:20.299354    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:48:24.299344403 +0000 UTC m=+42.620261675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:20.345146 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:20.345095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:20.642905 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:20.642864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:20.743572 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:20.743534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:22.383228 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:22.383122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:24.337338 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:24.337279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:24.337338 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:24.337343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:24.337402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:24.337468    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:24.337518    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:24.337540    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:32.337520044 +0000 UTC m=+50.658437314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:24.337559    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:48:32.33755002 +0000 UTC m=+50.658467285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:24.337592    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:24.337602    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:48:24.337922 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:24.337629    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:48:32.337619881 +0000 UTC m=+50.658537144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:48:25.364249 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.364190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x8n46" event={"ID":"1947ce46-e2e0-45af-ae40-738b891e5894","Type":"ContainerStarted","Data":"ac6d5c357cbb39ba5e2bd129e936e37239ed6c36aea299e2e8f1cc712a22e35a"}
Jun 15 05:48:25.368827 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.368706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" event={"ID":"1345c494-9ebb-4787-b869-3fd018e003b7","Type":"ContainerStarted","Data":"8caeb318394f307f056872d98357988ef214addf4d86d02a85be76c676a070e8"}
Jun 15 05:48:25.369644 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.369595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:25.370763 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.370706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:48:25.371561 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.371541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" event={"ID":"1dfed647-4879-412d-82ac-c256224b3d0c","Type":"ContainerStarted","Data":"0483d7a7d8deb8e735942bd3774df45e14d906962c69b095a3e2f0201a988d8b"}
Jun 15 05:48:25.374426 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.374360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" event={"ID":"b9c0bd7a-8b7d-401b-a301-d4843626a780","Type":"ContainerStarted","Data":"7e42a39c7d0184da222bdfcb633769875153ccd5cf9ac4737459169752d52c5b"}
Jun 15 05:48:25.381017 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.380965    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x8n46" podStartSLOduration=1.364561773 podStartE2EDuration="9.38094452s" totalImagesPullingTime="8.016382747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:25.380066044 +0000 UTC m=+43.700983331" watchObservedRunningTime="2026-06-15 05:48:25.38094452 +0000 UTC m=+43.701861806"
Jun 15 05:48:25.395916 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.394954    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" podStartSLOduration=5.347819068 podStartE2EDuration="13.394931298s" totalImagesPullingTime="8.04711223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:25.394638074 +0000 UTC m=+43.715555361" watchObservedRunningTime="2026-06-15 05:48:25.394931298 +0000 UTC m=+43.715848581"
Jun 15 05:48:25.413092 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:25.413037    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" podStartSLOduration=5.354440626 podStartE2EDuration="13.413020737s" totalImagesPullingTime="8.058580111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:25.412932055 +0000 UTC m=+43.733849343" watchObservedRunningTime="2026-06-15 05:48:25.413020737 +0000 UTC m=+43.733938023"
Jun 15 05:48:26.379479 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:26.379440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" event={"ID":"1dfed647-4879-412d-82ac-c256224b3d0c","Type":"ContainerStarted","Data":"aa16eb25397ee5766e750a4b07e524cb62e2169b04190d773b84423925e91551"}
Jun 15 05:48:26.379939 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:26.379491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" event={"ID":"1dfed647-4879-412d-82ac-c256224b3d0c","Type":"ContainerStarted","Data":"e88f98da0e2b6607c63693a59c91961fcbfaadbdc2f8a8cb4e27d35ca88e90a0"}
Jun 15 05:48:26.398768 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:26.398689    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" podStartSLOduration=6.338799448 podStartE2EDuration="14.398673826s" totalImagesPullingTime="8.059874378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:26.397467237 +0000 UTC m=+44.718384522" watchObservedRunningTime="2026-06-15 05:48:26.398673826 +0000 UTC m=+44.719591113"
Jun 15 05:48:26.761482 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:26.761394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:26.813323 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:26.813291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:30.644060 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:30.644021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:30.706040 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:30.706002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:30.978222 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:30.978141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:31.027294 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:31.027254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:32.405146 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:32.405112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:32.405146 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:32.405152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:32.405208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:32.405274    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:32.405324    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:32.405333    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:32.405350    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:48.405328829 +0000 UTC m=+66.726246117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:32.405367    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:48:48.405359378 +0000 UTC m=+66.726276650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:32.405350    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:48:32.405610 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:32.405481    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:48:48.405459351 +0000 UTC m=+66.726376633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:48:40.240038 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:40.240001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:40.373305 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:40.373274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:40.475027 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:40.474991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:40.529367 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:40.529286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:41.318623 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:41.318588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:41.379562 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:41.379526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:42.315895 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:42.315863    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnn2f"
Jun 15 05:48:42.468182 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:42.468148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:42.570942 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:42.570864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:47.926188 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:47.926142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:47.928704 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:47.928683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:48:47.940097 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:47.940066    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/369df3e2-42a9-4674-a0a9-3ed49caa5dc4-original-pull-secret\") pod \"global-pull-secret-syncer-7qgzl\" (UID: \"369df3e2-42a9-4674-a0a9-3ed49caa5dc4\") " pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:48.027520 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.027471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:48:48.030111 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.030094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:48:48.037842 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.037822    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:48:48.037903 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.037884    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:52.03786765 +0000 UTC m=+130.358784914 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : secret "metrics-daemon-secret" not found
Jun 15 05:48:48.128279 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.128241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:48.130957 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.130936    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:48:48.140820 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.140798    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:48:48.152021 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.151991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzgh\" (UniqueName: \"kubernetes.io/projected/915e92a0-de1d-46ad-ba10-bcab485dc2ad-kube-api-access-bnzgh\") pod \"network-check-target-thbgq\" (UID: \"915e92a0-de1d-46ad-ba10-bcab485dc2ad\") " pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:48.179085 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.178991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qfkm9\""
Jun 15 05:48:48.186734 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.186683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:48.191403 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.191384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7qgzl"
Jun 15 05:48:48.316163 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.316121    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-thbgq"]
Jun 15 05:48:48.320174 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:48:48.320141    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915e92a0_de1d_46ad_ba10_bcab485dc2ad.slice/crio-1e545f507e0bbc4014d5c565683c22cf62cd62a0a37a10056c644d91ab308704 WatchSource:0}: Error finding container 1e545f507e0bbc4014d5c565683c22cf62cd62a0a37a10056c644d91ab308704: Status 404 returned error can't find the container with id 1e545f507e0bbc4014d5c565683c22cf62cd62a0a37a10056c644d91ab308704
Jun 15 05:48:48.333575 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.333549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7qgzl"]
Jun 15 05:48:48.336220 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:48:48.336190    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369df3e2_42a9_4674_a0a9_3ed49caa5dc4.slice/crio-0f9e093e01a0d5a59b8abd839ff5450653c6c1337fdda0fbd1fa6acfb1e571c3 WatchSource:0}: Error finding container 0f9e093e01a0d5a59b8abd839ff5450653c6c1337fdda0fbd1fa6acfb1e571c3: Status 404 returned error can't find the container with id 0f9e093e01a0d5a59b8abd839ff5450653c6c1337fdda0fbd1fa6acfb1e571c3
Jun 15 05:48:48.431140 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.431052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:48:48.431318 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.431137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:48:48.431318 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.431168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:48:48.431318 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.431223    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:48.431318 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.431275    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:48:48.431318 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.431289    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:49:20.431272507 +0000 UTC m=+98.752189771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:48.431518 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.431325    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:20.431312427 +0000 UTC m=+98.752229691 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:48:48.431518 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.431281    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:48:48.431518 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.431340    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:48:48.431518 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:48:48.431362    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:49:20.431354136 +0000 UTC m=+98.752271400 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:48:48.443610 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.443573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7qgzl" event={"ID":"369df3e2-42a9-4674-a0a9-3ed49caa5dc4","Type":"ContainerStarted","Data":"0f9e093e01a0d5a59b8abd839ff5450653c6c1337fdda0fbd1fa6acfb1e571c3"}
Jun 15 05:48:48.444672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:48.444650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-thbgq" event={"ID":"915e92a0-de1d-46ad-ba10-bcab485dc2ad","Type":"ContainerStarted","Data":"1e545f507e0bbc4014d5c565683c22cf62cd62a0a37a10056c644d91ab308704"}
Jun 15 05:48:49.494402 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:49.494364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:52.647142 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:52.647102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:52.701954 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:52.701921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:53.465571 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:53.465537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7qgzl" event={"ID":"369df3e2-42a9-4674-a0a9-3ed49caa5dc4","Type":"ContainerStarted","Data":"7efe19f194211452ebf6f111e7f4b48af23dce09c3d8e3916de2df5245d8400c"}
Jun 15 05:48:53.467116 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:53.467088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-thbgq" event={"ID":"915e92a0-de1d-46ad-ba10-bcab485dc2ad","Type":"ContainerStarted","Data":"b05a0d858f49a8f33c4a76d7558df816d6219d677cd8c9fc9a625a692a6dd992"}
Jun 15 05:48:53.467232 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:53.467161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:48:53.480569 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:53.480518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7qgzl" podStartSLOduration=66.859679385 podStartE2EDuration="1m11.480501454s" totalImagesPullingTime="4.620822069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:53.47976915 +0000 UTC m=+71.800686437" watchObservedRunningTime="2026-06-15 05:48:53.480501454 +0000 UTC m=+71.801418740"
Jun 15 05:48:53.493215 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:53.493171    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-thbgq" podStartSLOduration=66.855777613 podStartE2EDuration="1m11.493155103s" totalImagesPullingTime="4.63737749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:53.493121067 +0000 UTC m=+71.814038356" watchObservedRunningTime="2026-06-15 05:48:53.493155103 +0000 UTC m=+71.814072366"
Jun 15 05:48:54.400356 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:54.400313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:54.456122 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:54.456082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:56.673075 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:56.672967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:56.728102 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:56.728067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:57.888146 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:57.888109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:48:57.988742 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:48:57.988691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:49:18.351636 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:49:18.351604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:49:18.410008 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:49:18.409962    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:49:20.488939 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:49:20.488837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:49:20.488939 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:49:20.488890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:49:20.488939 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:49:20.488921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:49:20.489359 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:20.488961    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:49:20.489359 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:20.489002    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:49:20.489359 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:20.489011    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:49:20.489359 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:20.489028    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:49:20.489359 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:20.489051    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:50:24.489033052 +0000 UTC m=+162.809950335 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:49:20.489359 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:20.489068    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:50:24.4890613 +0000 UTC m=+162.809978563 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:49:20.489359 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:20.489079    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:50:24.489074271 +0000 UTC m=+162.809991534 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:49:24.473202 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:49:24.473166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-thbgq"
Jun 15 05:49:52.124942 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:49:52.124881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:49:52.125513 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:52.125058    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:49:52.125513 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:49:52.125146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:54.12512524 +0000 UTC m=+252.446042519 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : secret "metrics-daemon-secret" not found
Jun 15 05:50:19.561994 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:19.561930    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf" podUID="415c4a6f-c939-42f7-9c11-c1730085b403"
Jun 15 05:50:19.589424 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:19.589391    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-757f768787-b5jx5" podUID="296f149e-037e-4f23-91d4-d248e11e20ee"
Jun 15 05:50:19.623248 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:19.623201    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zpd25" podUID="5a981072-5ef0-49c2-a80d-1dc2dcc4488d"
Jun 15 05:50:19.718986 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:19.718958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:50:19.719124 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:19.718965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zpd25"
Jun 15 05:50:19.719164 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:19.718962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:50:21.201372 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:21.201317    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:50:24.570387 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:24.570337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:50:24.570387 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:24.570387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:24.570439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:24.570494    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:24.570550    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:24.570553    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:24.570573    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757f768787-b5jx5: secret "image-registry-tls" not found
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:24.570562    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert podName:415c4a6f-c939-42f7-9c11-c1730085b403 nodeName:}" failed. No retries permitted until 2026-06-15 05:52:26.57054185 +0000 UTC m=+284.891459133 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert") pod "networking-console-plugin-867b76cb74-gvhjf" (UID: "415c4a6f-c939-42f7-9c11-c1730085b403") : secret "networking-console-plugin-cert" not found
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:24.570638    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls podName:5a981072-5ef0-49c2-a80d-1dc2dcc4488d nodeName:}" failed. No retries permitted until 2026-06-15 05:52:26.570625706 +0000 UTC m=+284.891542970 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls") pod "dns-default-zpd25" (UID: "5a981072-5ef0-49c2-a80d-1dc2dcc4488d") : secret "dns-default-metrics-tls" not found
Jun 15 05:50:24.570905 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:50:24.570649    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls podName:296f149e-037e-4f23-91d4-d248e11e20ee nodeName:}" failed. No retries permitted until 2026-06-15 05:52:26.570643538 +0000 UTC m=+284.891560803 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls") pod "image-registry-757f768787-b5jx5" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee") : secret "image-registry-tls" not found
Jun 15 05:50:25.369908 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:25.369868    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" podUID="1345c494-9ebb-4787-b869-3fd018e003b7" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
Jun 15 05:50:25.738699 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:25.738615    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9c0bd7a-8b7d-401b-a301-d4843626a780" containerID="7e42a39c7d0184da222bdfcb633769875153ccd5cf9ac4737459169752d52c5b" exitCode=255
Jun 15 05:50:25.738699 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:25.738690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" event={"ID":"b9c0bd7a-8b7d-401b-a301-d4843626a780","Type":"ContainerDied","Data":"7e42a39c7d0184da222bdfcb633769875153ccd5cf9ac4737459169752d52c5b"}
Jun 15 05:50:25.739190 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:25.739023    2579 scope.go:117] "RemoveContainer" containerID="7e42a39c7d0184da222bdfcb633769875153ccd5cf9ac4737459169752d52c5b"
Jun 15 05:50:25.740182 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:25.740159    2579 generic.go:356] "Generic (PLEG): container finished" podID="1345c494-9ebb-4787-b869-3fd018e003b7" containerID="8caeb318394f307f056872d98357988ef214addf4d86d02a85be76c676a070e8" exitCode=1
Jun 15 05:50:25.740262 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:25.740195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" event={"ID":"1345c494-9ebb-4787-b869-3fd018e003b7","Type":"ContainerDied","Data":"8caeb318394f307f056872d98357988ef214addf4d86d02a85be76c676a070e8"}
Jun 15 05:50:25.740523 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:25.740507    2579 scope.go:117] "RemoveContainer" containerID="8caeb318394f307f056872d98357988ef214addf4d86d02a85be76c676a070e8"
Jun 15 05:50:26.745250 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:26.745154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" event={"ID":"b9c0bd7a-8b7d-401b-a301-d4843626a780","Type":"ContainerStarted","Data":"03ffa88d78862594d17e76769b3b3466385f9bb48d6a00493fe063ebac112bf4"}
Jun 15 05:50:26.746751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:26.746708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" event={"ID":"1345c494-9ebb-4787-b869-3fd018e003b7","Type":"ContainerStarted","Data":"00ac080fb5b17d2bae4397104be82a5b1adf7655116d698808c00beab702e45e"}
Jun 15 05:50:26.747017 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:26.746999    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:50:26.747603 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:26.747586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:50:34.154308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:50:34.154265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:51:33.932444 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:51:33.932408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:51:40.282155 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:51:40.282121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:51:54.154462 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:51:54.154421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:51:54.155136 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:51:54.154537    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:51:54.155136 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:51:54.154597    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs podName:9a3746b9-06bd-418a-b727-fcc71c98d786 nodeName:}" failed. No retries permitted until 2026-06-15 05:53:56.15458305 +0000 UTC m=+374.475500315 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs") pod "network-metrics-daemon-smsbk" (UID: "9a3746b9-06bd-418a-b727-fcc71c98d786") : secret "metrics-daemon-secret" not found
Jun 15 05:51:58.077158 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:51:58.077066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:51:58.129955 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:51:58.129925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:05.510368 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:05.510329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:05.611220 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:05.611186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:07.127176 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:07.127139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:07.227909 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:07.227875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:08.894075 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:08.894038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:08.996784 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:08.996753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:09.156238 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.156157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:09.206842 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.206807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:09.601057 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.600976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:09.707761 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.707707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:09.868605 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.868569    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g7xhw"]
Jun 15 05:52:09.871598 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.871575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:09.873990 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.873965    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 15 05:52:09.874163 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.873994    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 15 05:52:09.874270 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.874254    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7z2gq\""
Jun 15 05:52:09.874879 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.874861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 15 05:52:09.875096 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.875082    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 15 05:52:09.885844 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.885484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g7xhw"]
Jun 15 05:52:09.926837 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.926803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:09.973081 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.973040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:09.973252 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.973092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-data-volume\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:09.973252 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.973115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-crio-socket\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:09.973252 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.973148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:09.973252 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:09.973218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbzs\" (UniqueName: \"kubernetes.io/projected/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-api-access-kgbzs\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.053447 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.053414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:10.074581 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.074553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbzs\" (UniqueName: \"kubernetes.io/projected/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-api-access-kgbzs\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.074718 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.074601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.074718 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.074620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-data-volume\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.074857 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.074722    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:10.074857 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.074773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-crio-socket\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.074857 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.074821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:10.574799433 +0000 UTC m=+268.895716710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:10.075002 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.074866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.075002 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.074878    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-crio-socket\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.075002 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.074948    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-data-volume\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.075002 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.074977    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:52:10.075126 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.075024    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:10.575015239 +0000 UTC m=+268.895932502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : configmap "kube-rbac-proxy" not found
Jun 15 05:52:10.082830 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.082811    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbzs\" (UniqueName: \"kubernetes.io/projected/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-api-access-kgbzs\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.178014 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.177925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:10.272089 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.272052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:10.373698 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.373667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:10.580035 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.579946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.580035 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:10.580029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:10.580224 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.580077    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:52:10.580224 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.580122    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:10.580224 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.580162    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:11.580144794 +0000 UTC m=+269.901062061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : configmap "kube-rbac-proxy" not found
Jun 15 05:52:10.580224 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:10.580182    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:11.580170997 +0000 UTC m=+269.901088262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:11.238055 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:11.238021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:11.589950 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:11.589859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:11.590119 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:11.589952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:11.590119 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:11.589980    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:52:11.590119 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:11.590044    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:13.59002863 +0000 UTC m=+271.910945894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : configmap "kube-rbac-proxy" not found
Jun 15 05:52:11.590119 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:11.590073    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:11.590275 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:11.590124    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:13.590112793 +0000 UTC m=+271.911030057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:12.069642 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:12.069612    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcp6n_d31e44b4-6f88-42da-b187-34d230a5ded0/dns-node-resolver/0.log"
Jun 15 05:52:12.168245 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:12.168211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:12.669064 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:12.669039    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2vs2q_11b70477-ceae-4d96-a015-b52d5659b1a6/node-ca/0.log"
Jun 15 05:52:13.606116 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:13.606077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:13.606299 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:13.606139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:13.606299 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:13.606224    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:13.606299 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:13.606234    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:52:13.606299 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:13.606287    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:17.606273798 +0000 UTC m=+275.927191063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : configmap "kube-rbac-proxy" not found
Jun 15 05:52:13.606299 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:13.606299    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:17.606293431 +0000 UTC m=+275.927210695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:13.859268 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:13.859182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:13.868672 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:13.868648    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x8n46_1947ce46-e2e0-45af-ae40-738b891e5894/serve-healthcheck-canary/0.log"
Jun 15 05:52:14.022939 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:14.022907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:14.124555 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:14.124462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:14.219875 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:14.219843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:14.284788 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:14.284752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:14.447039 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:14.446951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:14.555197 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:14.555165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:15.279473 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:15.279440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:15.586261 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:15.586185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:15.953057 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:15.953027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:16.166893 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:16.166861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:16.382669 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:16.382643    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:16.556419 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:16.556387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:17.467940 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:17.467906    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:17.568580 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:17.568549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:17.641025 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:17.640989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:17.641177 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:17.641046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:17.641177 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:17.641133    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:17.641177 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:17.641150    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:52:17.641276 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:17.641198    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:25.641181627 +0000 UTC m=+283.962098890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:52:17.641276 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:17.641212    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm podName:2fa8a952-44a1-48d5-90cf-cbd7966d5cac nodeName:}" failed. No retries permitted until 2026-06-15 05:52:25.641205112 +0000 UTC m=+283.962122375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7xhw" (UID: "2fa8a952-44a1-48d5-90cf-cbd7966d5cac") : configmap "kube-rbac-proxy" not found
Jun 15 05:52:17.775751 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:17.775653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:17.831008 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:17.830977    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:18.157961 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:18.157925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:18.355449 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:18.355419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:18.555345 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:18.555267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:18.654464 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:18.654436    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:19.553131 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:19.553097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:19.962032 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:19.961999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:20.155939 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:20.155897    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:20.350156 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:20.350055    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:20.408715 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:20.408679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:20.754260 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:20.754221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:21.352346 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:21.352305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:21.754063 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:21.754026    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:22.176921 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:22.176885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:22.235098 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:22.235070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:22.354850 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:22.354820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:22.719956 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:22.719917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf" podUID="415c4a6f-c939-42f7-9c11-c1730085b403"
Jun 15 05:52:22.719956 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:22.719944    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zpd25" podUID="5a981072-5ef0-49c2-a80d-1dc2dcc4488d"
Jun 15 05:52:22.720144 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:22.719994    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-757f768787-b5jx5" podUID="296f149e-037e-4f23-91d4-d248e11e20ee"
Jun 15 05:52:22.958048 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:22.958011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:23.083153 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.083080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:52:23.083280 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.083090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zpd25"
Jun 15 05:52:23.083280 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.083224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:52:23.258329 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.258297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:23.359335 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.359301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:23.459745 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.459699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:23.510973 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.510946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:23.763342 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.763256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:23.957872 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:23.957839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:24.382886 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:24.382856    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:24.762262 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:24.762178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:24.956879 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:24.956836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:25.710849 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:25.710809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:25.711308 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:25.710869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:25.711352 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:25.711337    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:25.713259 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:25.713238    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fa8a952-44a1-48d5-90cf-cbd7966d5cac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7xhw\" (UID: \"2fa8a952-44a1-48d5-90cf-cbd7966d5cac\") " pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:25.780508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:25.780471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g7xhw"
Jun 15 05:52:25.910980 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:25.910956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g7xhw"]
Jun 15 05:52:25.922692 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:52:25.922664    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa8a952_44a1_48d5_90cf_cbd7966d5cac.slice/crio-a12bbdbc9754f21332b311f525def885e2732b850e9a98c2d3ec63e54deb0024 WatchSource:0}: Error finding container a12bbdbc9754f21332b311f525def885e2732b850e9a98c2d3ec63e54deb0024: Status 404 returned error can't find the container with id a12bbdbc9754f21332b311f525def885e2732b850e9a98c2d3ec63e54deb0024
Jun 15 05:52:26.093063 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.092968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7xhw" event={"ID":"2fa8a952-44a1-48d5-90cf-cbd7966d5cac","Type":"ContainerStarted","Data":"a12bbdbc9754f21332b311f525def885e2732b850e9a98c2d3ec63e54deb0024"}
Jun 15 05:52:26.094621 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.094596    2579 generic.go:356] "Generic (PLEG): container finished" podID="1345c494-9ebb-4787-b869-3fd018e003b7" containerID="00ac080fb5b17d2bae4397104be82a5b1adf7655116d698808c00beab702e45e" exitCode=1
Jun 15 05:52:26.094780 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.094675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" event={"ID":"1345c494-9ebb-4787-b869-3fd018e003b7","Type":"ContainerDied","Data":"00ac080fb5b17d2bae4397104be82a5b1adf7655116d698808c00beab702e45e"}
Jun 15 05:52:26.094780 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.094756    2579 scope.go:117] "RemoveContainer" containerID="8caeb318394f307f056872d98357988ef214addf4d86d02a85be76c676a070e8"
Jun 15 05:52:26.095079 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.095062    2579 scope.go:117] "RemoveContainer" containerID="00ac080fb5b17d2bae4397104be82a5b1adf7655116d698808c00beab702e45e"
Jun 15 05:52:26.095270 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:26.095242    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-7d5cbb759-qcmzx_open-cluster-management-agent-addon(1345c494-9ebb-4787-b869-3fd018e003b7)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" podUID="1345c494-9ebb-4787-b869-3fd018e003b7"
Jun 15 05:52:26.096467 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.096446    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9c0bd7a-8b7d-401b-a301-d4843626a780" containerID="03ffa88d78862594d17e76769b3b3466385f9bb48d6a00493fe063ebac112bf4" exitCode=255
Jun 15 05:52:26.096551 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.096476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" event={"ID":"b9c0bd7a-8b7d-401b-a301-d4843626a780","Type":"ContainerDied","Data":"03ffa88d78862594d17e76769b3b3466385f9bb48d6a00493fe063ebac112bf4"}
Jun 15 05:52:26.096772 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.096757    2579 scope.go:117] "RemoveContainer" containerID="03ffa88d78862594d17e76769b3b3466385f9bb48d6a00493fe063ebac112bf4"
Jun 15 05:52:26.096894 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:26.096879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7d75854d66-2b8wk_open-cluster-management-agent-addon(b9c0bd7a-8b7d-401b-a301-d4843626a780)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" podUID="b9c0bd7a-8b7d-401b-a301-d4843626a780"
Jun 15 05:52:26.105183 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.105168    2579 scope.go:117] "RemoveContainer" containerID="7e42a39c7d0184da222bdfcb633769875153ccd5cf9ac4737459169752d52c5b"
Jun 15 05:52:26.618797 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.618759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:52:26.618998 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.618862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:52:26.618998 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.618898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:52:26.622258 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.622231    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"image-registry-757f768787-b5jx5\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") " pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:52:26.624373 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.624342    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a981072-5ef0-49c2-a80d-1dc2dcc4488d-metrics-tls\") pod \"dns-default-zpd25\" (UID: \"5a981072-5ef0-49c2-a80d-1dc2dcc4488d\") " pod="openshift-dns/dns-default-zpd25"
Jun 15 05:52:26.624838 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.624814    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/415c4a6f-c939-42f7-9c11-c1730085b403-networking-console-plugin-cert\") pod \"networking-console-plugin-867b76cb74-gvhjf\" (UID: \"415c4a6f-c939-42f7-9c11-c1730085b403\") " pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:52:26.687253 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.687218    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d82dw\""
Jun 15 05:52:26.687440 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.687257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j676s\""
Jun 15 05:52:26.687440 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.687426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8k6cb\""
Jun 15 05:52:26.695141 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.695108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zpd25"
Jun 15 05:52:26.695292 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.695212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:52:26.695292 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.695278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"
Jun 15 05:52:26.747679 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.747570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:52:26.760396 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.760361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:26.813174 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.813135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:26.883186 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.883094    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk"
Jun 15 05:52:26.895676 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.895638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zpd25"]
Jun 15 05:52:26.918540 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.918507    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:52:26.923555 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:26.923525    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-867b76cb74-gvhjf"]
Jun 15 05:52:26.927354 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:52:26.927314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415c4a6f_c939_42f7_9c11_c1730085b403.slice/crio-8dcb8981f352ed8fc65f6d4a0fcd55c2b1e75d90a65ed0ef785f67f17a89686d WatchSource:0}: Error finding container 8dcb8981f352ed8fc65f6d4a0fcd55c2b1e75d90a65ed0ef785f67f17a89686d: Status 404 returned error can't find the container with id 8dcb8981f352ed8fc65f6d4a0fcd55c2b1e75d90a65ed0ef785f67f17a89686d
Jun 15 05:52:27.101761 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:27.101706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf" event={"ID":"415c4a6f-c939-42f7-9c11-c1730085b403","Type":"ContainerStarted","Data":"8dcb8981f352ed8fc65f6d4a0fcd55c2b1e75d90a65ed0ef785f67f17a89686d"}
Jun 15 05:52:27.103811 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:27.103787    2579 scope.go:117] "RemoveContainer" containerID="00ac080fb5b17d2bae4397104be82a5b1adf7655116d698808c00beab702e45e"
Jun 15 05:52:27.104024 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:27.103989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-7d5cbb759-qcmzx_open-cluster-management-agent-addon(1345c494-9ebb-4787-b869-3fd018e003b7)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" podUID="1345c494-9ebb-4787-b869-3fd018e003b7"
Jun 15 05:52:27.105077 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:27.105014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zpd25" event={"ID":"5a981072-5ef0-49c2-a80d-1dc2dcc4488d","Type":"ContainerStarted","Data":"85b901e21e96c52a44eaa1bcf474114120cfddd6b08d08fd0c691734e155652e"}
Jun 15 05:52:27.107051 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:27.107029    2579 scope.go:117] "RemoveContainer" containerID="03ffa88d78862594d17e76769b3b3466385f9bb48d6a00493fe063ebac112bf4"
Jun 15 05:52:27.107231 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:27.107204    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7d75854d66-2b8wk_open-cluster-management-agent-addon(b9c0bd7a-8b7d-401b-a301-d4843626a780)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" podUID="b9c0bd7a-8b7d-401b-a301-d4843626a780"
Jun 15 05:52:27.128045 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:27.128014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757f768787-b5jx5"]
Jun 15 05:52:27.132053 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:52:27.132021    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296f149e_037e_4f23_91d4_d248e11e20ee.slice/crio-ac5db3e07dfab9f047b6d63dabf50c611988826ff572f061dfc971c744c110c5 WatchSource:0}: Error finding container ac5db3e07dfab9f047b6d63dabf50c611988826ff572f061dfc971c744c110c5: Status 404 returned error can't find the container with id ac5db3e07dfab9f047b6d63dabf50c611988826ff572f061dfc971c744c110c5
Jun 15 05:52:28.112094 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:28.112059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757f768787-b5jx5" event={"ID":"296f149e-037e-4f23-91d4-d248e11e20ee","Type":"ContainerStarted","Data":"02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562"}
Jun 15 05:52:28.112561 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:28.112166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757f768787-b5jx5" event={"ID":"296f149e-037e-4f23-91d4-d248e11e20ee","Type":"ContainerStarted","Data":"ac5db3e07dfab9f047b6d63dabf50c611988826ff572f061dfc971c744c110c5"}
Jun 15 05:52:28.112561 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:28.112255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:52:28.112561 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:28.112385    2579 scope.go:117] "RemoveContainer" containerID="00ac080fb5b17d2bae4397104be82a5b1adf7655116d698808c00beab702e45e"
Jun 15 05:52:28.112690 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:28.112560    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-7d5cbb759-qcmzx_open-cluster-management-agent-addon(1345c494-9ebb-4787-b869-3fd018e003b7)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" podUID="1345c494-9ebb-4787-b869-3fd018e003b7"
Jun 15 05:52:28.132458 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:28.132406    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-757f768787-b5jx5" podStartSLOduration=286.132389032 podStartE2EDuration="4m46.132389032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:28.131302401 +0000 UTC m=+286.452219686" watchObservedRunningTime="2026-06-15 05:52:28.132389032 +0000 UTC m=+286.453306320"
Jun 15 05:52:30.571943 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:30.571915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:30.672261 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:30.672177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:30.773024 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:30.772983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:30.876355 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:30.876323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:31.124888 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.124850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7xhw" event={"ID":"2fa8a952-44a1-48d5-90cf-cbd7966d5cac","Type":"ContainerStarted","Data":"90c65fa22ee7836dcf44040e0485a9cc2fd1324204a6f6d7c29060e11cbad06a"}
Jun 15 05:52:31.125053 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.124916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7xhw" event={"ID":"2fa8a952-44a1-48d5-90cf-cbd7966d5cac","Type":"ContainerStarted","Data":"5056c082d2df2d3097f5c2ba6dac6104420abb6bb2a34b34509c7f430616b905"}
Jun 15 05:52:31.126202 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.126167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf" event={"ID":"415c4a6f-c939-42f7-9c11-c1730085b403","Type":"ContainerStarted","Data":"e2b570062ecb81494a9779606e182e71c754b6efa94d938443ed4a31f97b8372"}
Jun 15 05:52:31.127846 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.127819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zpd25" event={"ID":"5a981072-5ef0-49c2-a80d-1dc2dcc4488d","Type":"ContainerStarted","Data":"b55dc3e65d091dec6a808a4bf12f9e307960e84a2dc195ceb4b79a463c718957"}
Jun 15 05:52:31.127964 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.127866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zpd25" event={"ID":"5a981072-5ef0-49c2-a80d-1dc2dcc4488d","Type":"ContainerStarted","Data":"99da11b109e3748b9518c59af7245fb52093f922c1c6c144e498cc857a3cecb9"}
Jun 15 05:52:31.127964 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.127900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zpd25"
Jun 15 05:52:31.141363 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.141300    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-867b76cb74-gvhjf" podStartSLOduration=269.713060875 podStartE2EDuration="4m33.141280668s" totalImagesPullingTime="3.428219793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:31.140886091 +0000 UTC m=+289.461803378" watchObservedRunningTime="2026-06-15 05:52:31.141280668 +0000 UTC m=+289.462197959"
Jun 15 05:52:31.157784 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.157735    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zpd25" podStartSLOduration=251.704714227 podStartE2EDuration="4m15.157706274s" totalImagesPullingTime="3.452992047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:31.156157887 +0000 UTC m=+289.477075173" watchObservedRunningTime="2026-06-15 05:52:31.157706274 +0000 UTC m=+289.478623559"
Jun 15 05:52:31.255810 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.255697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:31.307929 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.307896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:31.467964 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.467915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:31.656096 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.656056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:31.712038 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:31.712011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.056250 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.056167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.106450 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.106408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.242191 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.242152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.301334 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.301294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.550829 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.550790    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.652826 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.652669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.756696 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.756664    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:32.958639 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:32.958609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:33.062449 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:33.062416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:33.136623 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:33.136585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7xhw" event={"ID":"2fa8a952-44a1-48d5-90cf-cbd7966d5cac","Type":"ContainerStarted","Data":"7fb6a5a100e1039c49669113c24cf1f224248118fbc1716b9459fa8dba951de3"}
Jun 15 05:52:33.155998 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:33.155943    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g7xhw" podStartSLOduration=17.539212434 podStartE2EDuration="24.155928798s" totalImagesPullingTime="6.616716364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:52:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:33.155311268 +0000 UTC m=+291.476228565" watchObservedRunningTime="2026-06-15 05:52:33.155928798 +0000 UTC m=+291.476846084"
Jun 15 05:52:33.492937 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:33.492903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:34.483164 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:34.483126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:34.612920 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:34.612890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:34.775132 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:34.775048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:36.427296 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:36.427267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:37.155526 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:37.155490    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-smsbk" podUID="9a3746b9-06bd-418a-b727-fcc71c98d786"
Jun 15 05:52:38.669421 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.669391    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:38.879698 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.879664    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:38.944065 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.943978    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n68pr"]
Jun 15 05:52:38.948614 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.948594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:38.950809 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.950786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 15 05:52:38.950931 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.950814    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 15 05:52:38.950931 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.950814    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 15 05:52:38.951483 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.951468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qskhs\""
Jun 15 05:52:38.951483 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.951472    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 15 05:52:38.951627 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.951492    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 15 05:52:38.951627 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.951510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 15 05:52:38.977048 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:38.977017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.069089 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.069054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.118058 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-metrics-client-ca\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118236 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118236 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-textfile\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118236 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-tls\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118236 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-root\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118372 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-accelerators-collector-config\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118372 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-wtmp\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118372 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-sys\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.118372 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.118350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq6t\" (UniqueName: \"kubernetes.io/projected/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-kube-api-access-6jq6t\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.171058 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.171016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.219180 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-root\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219180 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-accelerators-collector-config\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219180 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-wtmp\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219180 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-sys\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219193    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-root\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq6t\" (UniqueName: \"kubernetes.io/projected/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-kube-api-access-6jq6t\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219245    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-sys\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-metrics-client-ca\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219319    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-wtmp\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-textfile\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-tls\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219943 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:39.219542    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jun 15 05:52:39.219943 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:52:39.219608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-tls podName:b35a58db-88d3-4da3-aee1-7b21a9cbf37a nodeName:}" failed. No retries permitted until 2026-06-15 05:52:39.719588085 +0000 UTC m=+298.040505354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-tls") pod "node-exporter-n68pr" (UID: "b35a58db-88d3-4da3-aee1-7b21a9cbf37a") : secret "node-exporter-tls" not found
Jun 15 05:52:39.219943 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219760    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-textfile\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219943 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219846    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-accelerators-collector-config\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.219943 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.219882    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-metrics-client-ca\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.221633 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.221613    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.229012 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.228991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq6t\" (UniqueName: \"kubernetes.io/projected/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-kube-api-access-6jq6t\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.273124 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.273085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.374669 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.374638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.477163 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.477074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.723983 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.723943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-tls\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.726310 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.726281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.726431 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.726362    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b35a58db-88d3-4da3-aee1-7b21a9cbf37a-node-exporter-tls\") pod \"node-exporter-n68pr\" (UID: \"b35a58db-88d3-4da3-aee1-7b21a9cbf37a\") " pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.857857 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.857827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n68pr"
Jun 15 05:52:39.871006 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:52:39.870977    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35a58db_88d3_4da3_aee1_7b21a9cbf37a.slice/crio-2148de7c3c5f0c29ec2547cd4a684a3e41390b2b9451797e4665e606623f68ef WatchSource:0}: Error finding container 2148de7c3c5f0c29ec2547cd4a684a3e41390b2b9451797e4665e606623f68ef: Status 404 returned error can't find the container with id 2148de7c3c5f0c29ec2547cd4a684a3e41390b2b9451797e4665e606623f68ef
Jun 15 05:52:39.904090 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.904053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:39.978708 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:39.978675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:40.154513 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:40.154429    2579 scope.go:117] "RemoveContainer" containerID="03ffa88d78862594d17e76769b3b3466385f9bb48d6a00493fe063ebac112bf4"
Jun 15 05:52:40.154513 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:40.154508    2579 scope.go:117] "RemoveContainer" containerID="00ac080fb5b17d2bae4397104be82a5b1adf7655116d698808c00beab702e45e"
Jun 15 05:52:40.158757 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:40.158713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n68pr" event={"ID":"b35a58db-88d3-4da3-aee1-7b21a9cbf37a","Type":"ContainerStarted","Data":"2148de7c3c5f0c29ec2547cd4a684a3e41390b2b9451797e4665e606623f68ef"}
Jun 15 05:52:40.193024 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:40.192998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:41.133156 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.133126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zpd25"
Jun 15 05:52:41.165288 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.165253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d75854d66-2b8wk" event={"ID":"b9c0bd7a-8b7d-401b-a301-d4843626a780","Type":"ContainerStarted","Data":"21455d9bee09677f30bde71f8d20a11bb55fe96880da39a3f4adf8025acb5693"}
Jun 15 05:52:41.167297 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.167266    2579 generic.go:356] "Generic (PLEG): container finished" podID="b35a58db-88d3-4da3-aee1-7b21a9cbf37a" containerID="e63eb494eebb1cd4790fa28400114adf930ac23c85f7646120fe04948ad83d03" exitCode=0
Jun 15 05:52:41.167441 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.167341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n68pr" event={"ID":"b35a58db-88d3-4da3-aee1-7b21a9cbf37a","Type":"ContainerDied","Data":"e63eb494eebb1cd4790fa28400114adf930ac23c85f7646120fe04948ad83d03"}
Jun 15 05:52:41.169359 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.169338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx" event={"ID":"1345c494-9ebb-4787-b869-3fd018e003b7","Type":"ContainerStarted","Data":"98dd94ea8b505b2a126b03dbccffcd89f42798a10b6483758fbe5cea7e8b23e8"}
Jun 15 05:52:41.169653 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.169615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:52:41.170902 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.170882    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5cbb759-qcmzx"
Jun 15 05:52:41.505460 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.505372    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:41.605881 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.605850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:41.928380 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.928346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:41.987316 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:41.987280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:42.109233 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:42.109202    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:52:42.109956 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:42.109938    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:52:42.116084 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:42.116066    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jun 15 05:52:42.144984 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:42.144860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:42.173994 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:42.173960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n68pr" event={"ID":"b35a58db-88d3-4da3-aee1-7b21a9cbf37a","Type":"ContainerStarted","Data":"59d1abec412b57858cdad02933158cdb4b09f0108e92d77ea6965827cd7b8889"}
Jun 15 05:52:42.174139 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:42.174010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n68pr" event={"ID":"b35a58db-88d3-4da3-aee1-7b21a9cbf37a","Type":"ContainerStarted","Data":"89da44a1d6fe1eaa1baa26260e3dbbd703357bce4698468847ab69c710fe400c"}
Jun 15 05:52:42.346657 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:42.346568    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:43.651818 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:43.651766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:44.770247 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:44.770211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:45.221182 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:45.221149    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:45.501176 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:45.501080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:45.599117 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:45.599085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:46.375332 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:46.375263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:46.424035 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:46.423999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:46.699613 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:46.699535    2579 patch_prober.go:28] interesting pod/image-registry-757f768787-b5jx5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 15 05:52:46.699778 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:46.699611    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-757f768787-b5jx5" podUID="296f149e-037e-4f23-91d4-d248e11e20ee" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:52:49.120578 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:49.120550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:52:49.140647 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:49.140596    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n68pr" podStartSLOduration=10.207670719 podStartE2EDuration="11.140577212s" totalImagesPullingTime="867.829284ms" totalInitContainerRuntime="65.077209ms" isStatefulPod=true podCreationTimestamp="2026-06-15 05:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:42.198215934 +0000 UTC m=+300.519133219" watchObservedRunningTime="2026-06-15 05:52:49.140577212 +0000 UTC m=+307.461494498"
Jun 15 05:52:49.404243 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:49.404155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:49.501066 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:49.501032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:49.598898 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:49.598861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:50.267516 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:50.267481    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:50.366960 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:50.366927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:50.920104 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:50.920075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:51.020302 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:51.020273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:52.156478 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:52.156441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:52:53.152427 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:53.152394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:53.252996 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:53.252965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:54.184420 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:54.184381    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757f768787-b5jx5"]
Jun 15 05:52:54.282390 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:54.282355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:54.383358 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:54.383324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:54.482014 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:54.481927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:55.454429 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:55.454398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:55.554693 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:55.554657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:56.765709 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:56.765677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:56.828562 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:56.828529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:59.288501 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:59.288458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:52:59.390358 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:52:59.390327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:00.690947 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:00.690914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:00.791944 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:00.791914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:02.491987 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:02.491955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:02.545259 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:02.545230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:10.547919 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:10.547875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:10.606459 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:10.606423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:10.656915 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:10.656887    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:11.762299 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:11.762267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:11.815237 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:11.815203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:19.203904 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.203858    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-757f768787-b5jx5" podUID="296f149e-037e-4f23-91d4-d248e11e20ee" containerName="registry" containerID="cri-o://02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562" gracePeriod=30
Jun 15 05:53:19.446301 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.446271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:53:19.517911 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.517824    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-installation-pull-secrets\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.517911 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.517870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-bound-sa-token\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.517911 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.517895    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt7dd\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-kube-api-access-vt7dd\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.518176 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.518004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-image-registry-private-configuration\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.518176 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.518078    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.518176 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.518116    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-registry-certificates\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.518176 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.518161    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-trusted-ca\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.518386 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.518232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/296f149e-037e-4f23-91d4-d248e11e20ee-ca-trust-extracted\") pod \"296f149e-037e-4f23-91d4-d248e11e20ee\" (UID: \"296f149e-037e-4f23-91d4-d248e11e20ee\") "
Jun 15 05:53:19.518859 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.518782    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:53:19.518859 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.518799    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:53:19.520862 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.520830    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-kube-api-access-vt7dd" (OuterVolumeSpecName: "kube-api-access-vt7dd") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "kube-api-access-vt7dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:53:19.520980 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.520867    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:53:19.520980 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.520907    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:53:19.520980 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.520958    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:53:19.521129 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.521037    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:53:19.527056 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.527035    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296f149e-037e-4f23-91d4-d248e11e20ee-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "296f149e-037e-4f23-91d4-d248e11e20ee" (UID: "296f149e-037e-4f23-91d4-d248e11e20ee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:53:19.619288 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619247    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-trusted-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:19.619288 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619281    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/296f149e-037e-4f23-91d4-d248e11e20ee-ca-trust-extracted\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:19.619288 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619295    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-installation-pull-secrets\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:19.619532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619307    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-bound-sa-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:19.619532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619319    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt7dd\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-kube-api-access-vt7dd\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:19.619532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619331    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/296f149e-037e-4f23-91d4-d248e11e20ee-image-registry-private-configuration\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:19.619532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619344    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/296f149e-037e-4f23-91d4-d248e11e20ee-registry-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:19.619532 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:19.619355    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/296f149e-037e-4f23-91d4-d248e11e20ee-registry-certificates\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:53:20.245881 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.245847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:20.301527 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.301487    2579 generic.go:356] "Generic (PLEG): container finished" podID="296f149e-037e-4f23-91d4-d248e11e20ee" containerID="02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562" exitCode=0
Jun 15 05:53:20.301710 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.301539    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757f768787-b5jx5"
Jun 15 05:53:20.301710 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.301567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757f768787-b5jx5" event={"ID":"296f149e-037e-4f23-91d4-d248e11e20ee","Type":"ContainerDied","Data":"02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562"}
Jun 15 05:53:20.301710 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.301636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757f768787-b5jx5" event={"ID":"296f149e-037e-4f23-91d4-d248e11e20ee","Type":"ContainerDied","Data":"ac5db3e07dfab9f047b6d63dabf50c611988826ff572f061dfc971c744c110c5"}
Jun 15 05:53:20.301710 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.301659    2579 scope.go:117] "RemoveContainer" containerID="02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562"
Jun 15 05:53:20.309586 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.309568    2579 scope.go:117] "RemoveContainer" containerID="02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562"
Jun 15 05:53:20.309878 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:53:20.309859    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562\": container with ID starting with 02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562 not found: ID does not exist" containerID="02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562"
Jun 15 05:53:20.309958 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.309888    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562"} err="failed to get container status \"02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562\": rpc error: code = NotFound desc = could not find container \"02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562\": container with ID starting with 02a973efd3aeabff3f8c6045ae340ca6c053e985b9a3dac3ee970a3e18a51562 not found: ID does not exist"
Jun 15 05:53:20.319346 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.319315    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757f768787-b5jx5"]
Jun 15 05:53:20.325843 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:20.325818    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-757f768787-b5jx5"]
Jun 15 05:53:22.158268 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:22.158230    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296f149e-037e-4f23-91d4-d248e11e20ee" path="/var/lib/kubelet/pods/296f149e-037e-4f23-91d4-d248e11e20ee/volumes"
Jun 15 05:53:24.142407 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:24.142377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:24.201077 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:24.201044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:36.912621 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:36.912538    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" podUID="1dfed647-4879-412d-82ac-c256224b3d0c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 15 05:53:46.912704 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:46.912662    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" podUID="1dfed647-4879-412d-82ac-c256224b3d0c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 15 05:53:55.079158 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:55.079122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:55.180053 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:55.180021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:56.204119 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.204085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:53:56.206473 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.206450    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a3746b9-06bd-418a-b727-fcc71c98d786-metrics-certs\") pod \"network-metrics-daemon-smsbk\" (UID: \"9a3746b9-06bd-418a-b727-fcc71c98d786\") " pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:53:56.360071 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.360041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdlts\""
Jun 15 05:53:56.368759 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.368718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smsbk"
Jun 15 05:53:56.493163 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.493136    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-smsbk"]
Jun 15 05:53:56.495707 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:53:56.495674    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3746b9_06bd_418a_b727_fcc71c98d786.slice/crio-5633cd907979805b32101dfb018e7d941c532c4ed28eb59bbc6d756ffa7646c6 WatchSource:0}: Error finding container 5633cd907979805b32101dfb018e7d941c532c4ed28eb59bbc6d756ffa7646c6: Status 404 returned error can't find the container with id 5633cd907979805b32101dfb018e7d941c532c4ed28eb59bbc6d756ffa7646c6
Jun 15 05:53:56.497677 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.497652    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:53:56.912633 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.912588    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" podUID="1dfed647-4879-412d-82ac-c256224b3d0c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 15 05:53:56.912825 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.912677    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt"
Jun 15 05:53:56.913249 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.913225    2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"aa16eb25397ee5766e750a4b07e524cb62e2169b04190d773b84423925e91551"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jun 15 05:53:56.913386 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:56.913309    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" podUID="1dfed647-4879-412d-82ac-c256224b3d0c" containerName="service-proxy" containerID="cri-o://aa16eb25397ee5766e750a4b07e524cb62e2169b04190d773b84423925e91551" gracePeriod=30
Jun 15 05:53:57.426979 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:57.426952    2579 generic.go:356] "Generic (PLEG): container finished" podID="1dfed647-4879-412d-82ac-c256224b3d0c" containerID="aa16eb25397ee5766e750a4b07e524cb62e2169b04190d773b84423925e91551" exitCode=2
Jun 15 05:53:57.427343 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:57.427027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" event={"ID":"1dfed647-4879-412d-82ac-c256224b3d0c","Type":"ContainerDied","Data":"aa16eb25397ee5766e750a4b07e524cb62e2169b04190d773b84423925e91551"}
Jun 15 05:53:57.427343 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:57.427073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5b856bb6-cp4lt" event={"ID":"1dfed647-4879-412d-82ac-c256224b3d0c","Type":"ContainerStarted","Data":"3a3a74a402e498da623708c909998b3f19f68ac09f8f67e12fc1d1d43dd90dff"}
Jun 15 05:53:57.428508 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:57.428489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-smsbk" event={"ID":"9a3746b9-06bd-418a-b727-fcc71c98d786","Type":"ContainerStarted","Data":"5633cd907979805b32101dfb018e7d941c532c4ed28eb59bbc6d756ffa7646c6"}
Jun 15 05:53:58.434408 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:58.434367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-smsbk" event={"ID":"9a3746b9-06bd-418a-b727-fcc71c98d786","Type":"ContainerStarted","Data":"f540915bb6a9f22dc5851308587402740faf0a7ed2f2915e29728f89e3a161b3"}
Jun 15 05:53:58.434898 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:58.434418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-smsbk" event={"ID":"9a3746b9-06bd-418a-b727-fcc71c98d786","Type":"ContainerStarted","Data":"7d31497810b93b15914f85a30ae328daa099a5ef9186fc33e76bf9f4d709d2bf"}
Jun 15 05:53:58.452586 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:58.452531    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-smsbk" podStartSLOduration=375.597848394 podStartE2EDuration="6m16.4525158s" totalImagesPullingTime="854.667406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:53:58.451711024 +0000 UTC m=+376.772628309" watchObservedRunningTime="2026-06-15 05:53:58.4525158 +0000 UTC m=+376.773433102"
Jun 15 05:53:58.563905 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:58.563870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:58.617765 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:58.617717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:58.686262 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:58.686178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:53:59.900488 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:53:59.900442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:00.003863 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:00.003823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:00.103600 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:00.103570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:00.638800 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:00.638765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:00.739832 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:00.739794    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:06.882884 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:06.882843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:06.941602 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:06.941571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:07.822597 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:07.822531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:07.887678 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:07.887626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:10.424101 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:10.424068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:10.477302 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:10.477273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:10.525627 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:10.525590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:10.580665 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:10.580639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:11.771699 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:11.771650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:11.874655 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:11.874618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:20.537124 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:20.537034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:45.818259 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:45.818224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:45.866937 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:45.866909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:53.024542 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:53.024504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:53.133795 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:53.133764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:53.350837 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:53.350750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:53.448788 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:53.448747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:54:53.552699 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:54:53.552670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:19.967504 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:19.967469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:20.066205 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:20.066170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:24.779180 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:24.779141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:24.868112 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:24.868076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:25.002210 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:25.002145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:25.193988 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:25.193953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:25.485534 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:25.485446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:25.586603 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:25.586571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:25.689920 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:25.689882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:25.792256 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:25.792172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:26.096138 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:26.096057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:26.421102 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:26.421069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:26.676020 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:26.675939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:55:26.878766 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:55:26.878720    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:30.834464 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.834366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:30.905148 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.905117    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-xrr9r"]
Jun 15 05:56:30.905355 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.905343    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296f149e-037e-4f23-91d4-d248e11e20ee" containerName="registry"
Jun 15 05:56:30.905397 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.905356    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="296f149e-037e-4f23-91d4-d248e11e20ee" containerName="registry"
Jun 15 05:56:30.905435 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.905400    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="296f149e-037e-4f23-91d4-d248e11e20ee" containerName="registry"
Jun 15 05:56:30.908177 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.908160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:30.909029 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.909000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:30.911631 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.911593    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jun 15 05:56:30.912460 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.912441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 15 05:56:30.912460 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.912450    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-5x54c\""
Jun 15 05:56:30.912460 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.912462    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 15 05:56:30.920231 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.920197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-xrr9r"]
Jun 15 05:56:30.980392 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.980351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert\") pod \"kserve-controller-manager-6c7654bd99-xrr9r\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") " pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:30.980567 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.980399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8c7t\" (UniqueName: \"kubernetes.io/projected/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-kube-api-access-x8c7t\") pod \"kserve-controller-manager-6c7654bd99-xrr9r\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") " pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:30.985967 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:30.985933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:31.043034 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.043005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:31.081347 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.081313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert\") pod \"kserve-controller-manager-6c7654bd99-xrr9r\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") " pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:31.081428 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.081363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8c7t\" (UniqueName: \"kubernetes.io/projected/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-kube-api-access-x8c7t\") pod \"kserve-controller-manager-6c7654bd99-xrr9r\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") " pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:31.081475 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:56:31.081458    2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jun 15 05:56:31.081525 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:56:31.081517    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert podName:d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1 nodeName:}" failed. No retries permitted until 2026-06-15 05:56:31.581502113 +0000 UTC m=+529.902419377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert") pod "kserve-controller-manager-6c7654bd99-xrr9r" (UID: "d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1") : secret "kserve-webhook-server-cert" not found
Jun 15 05:56:31.092820 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.092748    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8c7t\" (UniqueName: \"kubernetes.io/projected/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-kube-api-access-x8c7t\") pod \"kserve-controller-manager-6c7654bd99-xrr9r\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") " pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:31.271299 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.271268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:31.553961 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.553929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:31.585303 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.585270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert\") pod \"kserve-controller-manager-6c7654bd99-xrr9r\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") " pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:31.587660 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.587632    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert\") pod \"kserve-controller-manager-6c7654bd99-xrr9r\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") " pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:31.655595 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.655567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:31.709867 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.709829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:56:31.818120 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.818036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:31.948001 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:31.947969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-xrr9r"]
Jun 15 05:56:31.951436 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:56:31.951407    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d9f405_0c6c_4aa3_9b24_1b6e2f1c2fb1.slice/crio-5b9b53f98fc3f0e8e8d5f33c7b3421b58a7de9b32604b673de1ffbff4cc0b4a3 WatchSource:0}: Error finding container 5b9b53f98fc3f0e8e8d5f33c7b3421b58a7de9b32604b673de1ffbff4cc0b4a3: Status 404 returned error can't find the container with id 5b9b53f98fc3f0e8e8d5f33c7b3421b58a7de9b32604b673de1ffbff4cc0b4a3
Jun 15 05:56:32.938133 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:32.938097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r" event={"ID":"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1","Type":"ContainerStarted","Data":"5b9b53f98fc3f0e8e8d5f33c7b3421b58a7de9b32604b673de1ffbff4cc0b4a3"}
Jun 15 05:56:34.945641 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:34.945605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r" event={"ID":"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1","Type":"ContainerStarted","Data":"d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52"}
Jun 15 05:56:34.946117 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:34.945766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:56:34.962040 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:56:34.961987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r" podStartSLOduration=2.633885796 podStartE2EDuration="4.961970411s" totalImagesPullingTime="2.328084615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:34.961817403 +0000 UTC m=+533.282734680" watchObservedRunningTime="2026-06-15 05:56:34.961970411 +0000 UTC m=+533.282887675"
Jun 15 05:57:05.954987 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:05.954951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:57:08.525513 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.525459    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-xrr9r"]
Jun 15 05:57:08.526032 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.525757    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r" podUID="d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" containerName="manager" containerID="cri-o://d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52" gracePeriod=10
Jun 15 05:57:08.553932 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.553905    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-d9hfx"]
Jun 15 05:57:08.558000 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.557979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.568887 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.568860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-d9hfx"]
Jun 15 05:57:08.748416 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.748382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817e29c6-b60a-48f8-97ad-72473abb589e-cert\") pod \"kserve-controller-manager-6c7654bd99-d9hfx\" (UID: \"817e29c6-b60a-48f8-97ad-72473abb589e\") " pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.748587 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.748424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbjs\" (UniqueName: \"kubernetes.io/projected/817e29c6-b60a-48f8-97ad-72473abb589e-kube-api-access-qrbjs\") pod \"kserve-controller-manager-6c7654bd99-d9hfx\" (UID: \"817e29c6-b60a-48f8-97ad-72473abb589e\") " pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.766353 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.766326    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:57:08.849142 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.849059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817e29c6-b60a-48f8-97ad-72473abb589e-cert\") pod \"kserve-controller-manager-6c7654bd99-d9hfx\" (UID: \"817e29c6-b60a-48f8-97ad-72473abb589e\") " pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.849142 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.849108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbjs\" (UniqueName: \"kubernetes.io/projected/817e29c6-b60a-48f8-97ad-72473abb589e-kube-api-access-qrbjs\") pod \"kserve-controller-manager-6c7654bd99-d9hfx\" (UID: \"817e29c6-b60a-48f8-97ad-72473abb589e\") " pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.851575 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.851545    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817e29c6-b60a-48f8-97ad-72473abb589e-cert\") pod \"kserve-controller-manager-6c7654bd99-d9hfx\" (UID: \"817e29c6-b60a-48f8-97ad-72473abb589e\") " pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.858850 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.858822    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbjs\" (UniqueName: \"kubernetes.io/projected/817e29c6-b60a-48f8-97ad-72473abb589e-kube-api-access-qrbjs\") pod \"kserve-controller-manager-6c7654bd99-d9hfx\" (UID: \"817e29c6-b60a-48f8-97ad-72473abb589e\") " pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.906119 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.906075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:08.950045 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.950010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert\") pod \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") "
Jun 15 05:57:08.950217 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.950072    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8c7t\" (UniqueName: \"kubernetes.io/projected/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-kube-api-access-x8c7t\") pod \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\" (UID: \"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1\") "
Jun 15 05:57:08.952286 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.952250    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert" (OuterVolumeSpecName: "cert") pod "d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" (UID: "d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:57:08.952286 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:08.952260    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-kube-api-access-x8c7t" (OuterVolumeSpecName: "kube-api-access-x8c7t") pod "d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" (UID: "d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1"). InnerVolumeSpecName "kube-api-access-x8c7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:57:09.032744 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.032691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-d9hfx"]
Jun 15 05:57:09.036256 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:57:09.036226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817e29c6_b60a_48f8_97ad_72473abb589e.slice/crio-8e188298c4304e05377d58296b14a0ad05742f988a35abcff0defa3733923cf6 WatchSource:0}: Error finding container 8e188298c4304e05377d58296b14a0ad05742f988a35abcff0defa3733923cf6: Status 404 returned error can't find the container with id 8e188298c4304e05377d58296b14a0ad05742f988a35abcff0defa3733923cf6
Jun 15 05:57:09.050676 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.050654    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:57:09.050676 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.050676    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8c7t\" (UniqueName: \"kubernetes.io/projected/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1-kube-api-access-x8c7t\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:57:09.059395 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.059363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx" event={"ID":"817e29c6-b60a-48f8-97ad-72473abb589e","Type":"ContainerStarted","Data":"8e188298c4304e05377d58296b14a0ad05742f988a35abcff0defa3733923cf6"}
Jun 15 05:57:09.060577 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.060550    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" containerID="d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52" exitCode=0
Jun 15 05:57:09.060679 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.060581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r" event={"ID":"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1","Type":"ContainerDied","Data":"d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52"}
Jun 15 05:57:09.060679 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.060612    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r"
Jun 15 05:57:09.060679 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.060628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-xrr9r" event={"ID":"d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1","Type":"ContainerDied","Data":"5b9b53f98fc3f0e8e8d5f33c7b3421b58a7de9b32604b673de1ffbff4cc0b4a3"}
Jun 15 05:57:09.060679 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.060650    2579 scope.go:117] "RemoveContainer" containerID="d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52"
Jun 15 05:57:09.068386 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.068357    2579 scope.go:117] "RemoveContainer" containerID="d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52"
Jun 15 05:57:09.068664 ip-10-0-135-167 kubenswrapper[2579]: E0615 05:57:09.068642    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52\": container with ID starting with d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52 not found: ID does not exist" containerID="d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52"
Jun 15 05:57:09.068782 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.068678    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52"} err="failed to get container status \"d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52\": rpc error: code = NotFound desc = could not find container \"d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52\": container with ID starting with d2ceb51ca961767fa3a7ae38bbcc1f050c845fd7d6c3a4ba6c5b7bf2df1f8d52 not found: ID does not exist"
Jun 15 05:57:09.081603 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.081575    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-xrr9r"]
Jun 15 05:57:09.085599 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:09.085574    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-xrr9r"]
Jun 15 05:57:10.065975 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:10.065936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx" event={"ID":"817e29c6-b60a-48f8-97ad-72473abb589e","Type":"ContainerStarted","Data":"ad11d1e720817cf1f28d07c021511fceedbd9b63d7f84f9a4312b5295a37af44"}
Jun 15 05:57:10.066429 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:10.066018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:10.084555 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:10.084496    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx" podStartSLOduration=1.753620535 podStartE2EDuration="2.084473781s" totalImagesPullingTime="330.853246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:10.084145284 +0000 UTC m=+568.405062569" watchObservedRunningTime="2026-06-15 05:57:10.084473781 +0000 UTC m=+568.405391066"
Jun 15 05:57:10.158294 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:10.158265    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" path="/var/lib/kubelet/pods/d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1/volumes"
Jun 15 05:57:13.287410 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:13.287374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:34.786998 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:34.786950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:34.835326 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:34.835297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:41.076087 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:41.076053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6c7654bd99-d9hfx"
Jun 15 05:57:42.011776 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.011708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:42.074659 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.074629    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-jmhw5"]
Jun 15 05:57:42.074889 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.074877    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" containerName="manager"
Jun 15 05:57:42.074944 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.074890    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" containerName="manager"
Jun 15 05:57:42.074944 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.074938    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d9f405-0c6c-4aa3-9b24-1b6e2f1c2fb1" containerName="manager"
Jun 15 05:57:42.077883 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.077864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.079845 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.079825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-nhvr8\""
Jun 15 05:57:42.079845 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.079825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jun 15 05:57:42.090200 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.090177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jmhw5"]
Jun 15 05:57:42.097860 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.097836    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-5fvck"]
Jun 15 05:57:42.097999 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.097853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:42.100903 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.100884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.102989 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.102952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-vm5z7\""
Jun 15 05:57:42.103208 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.103187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jun 15 05:57:42.113354 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.113330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-5fvck"]
Jun 15 05:57:42.131249 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.131223    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:57:42.131418 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.131319    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 05:57:42.166628 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.166521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312b2aac-d3e5-457a-aacc-4efbcb38f761-cert\") pod \"odh-model-controller-84b54b7f97-5fvck\" (UID: \"312b2aac-d3e5-457a-aacc-4efbcb38f761\") " pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.166628 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.166558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27n2\" (UniqueName: \"kubernetes.io/projected/312b2aac-d3e5-457a-aacc-4efbcb38f761-kube-api-access-v27n2\") pod \"odh-model-controller-84b54b7f97-5fvck\" (UID: \"312b2aac-d3e5-457a-aacc-4efbcb38f761\") " pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.166628 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.166586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5320aa-0600-481b-8e32-8c4cc5499b36-tls-certs\") pod \"model-serving-api-fd65d7d6-jmhw5\" (UID: \"6c5320aa-0600-481b-8e32-8c4cc5499b36\") " pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.175969 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.166660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4f9\" (UniqueName: \"kubernetes.io/projected/6c5320aa-0600-481b-8e32-8c4cc5499b36-kube-api-access-2j4f9\") pod \"model-serving-api-fd65d7d6-jmhw5\" (UID: \"6c5320aa-0600-481b-8e32-8c4cc5499b36\") " pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.208321 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.208276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:42.267093 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.267000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312b2aac-d3e5-457a-aacc-4efbcb38f761-cert\") pod \"odh-model-controller-84b54b7f97-5fvck\" (UID: \"312b2aac-d3e5-457a-aacc-4efbcb38f761\") " pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.267093 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.267044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27n2\" (UniqueName: \"kubernetes.io/projected/312b2aac-d3e5-457a-aacc-4efbcb38f761-kube-api-access-v27n2\") pod \"odh-model-controller-84b54b7f97-5fvck\" (UID: \"312b2aac-d3e5-457a-aacc-4efbcb38f761\") " pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.267093 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.267074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5320aa-0600-481b-8e32-8c4cc5499b36-tls-certs\") pod \"model-serving-api-fd65d7d6-jmhw5\" (UID: \"6c5320aa-0600-481b-8e32-8c4cc5499b36\") " pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.267539 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.267105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4f9\" (UniqueName: \"kubernetes.io/projected/6c5320aa-0600-481b-8e32-8c4cc5499b36-kube-api-access-2j4f9\") pod \"model-serving-api-fd65d7d6-jmhw5\" (UID: \"6c5320aa-0600-481b-8e32-8c4cc5499b36\") " pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.270198 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.270170    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5320aa-0600-481b-8e32-8c4cc5499b36-tls-certs\") pod \"model-serving-api-fd65d7d6-jmhw5\" (UID: \"6c5320aa-0600-481b-8e32-8c4cc5499b36\") " pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.270315 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.270228    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312b2aac-d3e5-457a-aacc-4efbcb38f761-cert\") pod \"odh-model-controller-84b54b7f97-5fvck\" (UID: \"312b2aac-d3e5-457a-aacc-4efbcb38f761\") " pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.278278 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.278251    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4f9\" (UniqueName: \"kubernetes.io/projected/6c5320aa-0600-481b-8e32-8c4cc5499b36-kube-api-access-2j4f9\") pod \"model-serving-api-fd65d7d6-jmhw5\" (UID: \"6c5320aa-0600-481b-8e32-8c4cc5499b36\") " pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.278391 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.278254    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27n2\" (UniqueName: \"kubernetes.io/projected/312b2aac-d3e5-457a-aacc-4efbcb38f761-kube-api-access-v27n2\") pod \"odh-model-controller-84b54b7f97-5fvck\" (UID: \"312b2aac-d3e5-457a-aacc-4efbcb38f761\") " pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.312165 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.312127    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:42.387014 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.386977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:42.411106 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.411076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:42.413711 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.413684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:42.468232 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.468126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:57:42.524542 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.524385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jmhw5"]
Jun 15 05:57:42.527416 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:57:42.527388    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5320aa_0600_481b_8e32_8c4cc5499b36.slice/crio-533d6b58f4e350533c98b88fafbb7212182a13daae96b8b2af768d5970adee96 WatchSource:0}: Error finding container 533d6b58f4e350533c98b88fafbb7212182a13daae96b8b2af768d5970adee96: Status 404 returned error can't find the container with id 533d6b58f4e350533c98b88fafbb7212182a13daae96b8b2af768d5970adee96
Jun 15 05:57:42.547951 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:42.547852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-5fvck"]
Jun 15 05:57:42.550264 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:57:42.550236    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312b2aac_d3e5_457a_aacc_4efbcb38f761.slice/crio-eb45b1de13051d6b5f0d7cfc70f16ee505c89d443722afa6090b3dc8743a3203 WatchSource:0}: Error finding container eb45b1de13051d6b5f0d7cfc70f16ee505c89d443722afa6090b3dc8743a3203: Status 404 returned error can't find the container with id eb45b1de13051d6b5f0d7cfc70f16ee505c89d443722afa6090b3dc8743a3203
Jun 15 05:57:43.178578 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:43.178535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jmhw5" event={"ID":"6c5320aa-0600-481b-8e32-8c4cc5499b36","Type":"ContainerStarted","Data":"533d6b58f4e350533c98b88fafbb7212182a13daae96b8b2af768d5970adee96"}
Jun 15 05:57:43.180086 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:43.180044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-5fvck" event={"ID":"312b2aac-d3e5-457a-aacc-4efbcb38f761","Type":"ContainerStarted","Data":"eb45b1de13051d6b5f0d7cfc70f16ee505c89d443722afa6090b3dc8743a3203"}
Jun 15 05:57:46.193473 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:46.193424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jmhw5" event={"ID":"6c5320aa-0600-481b-8e32-8c4cc5499b36","Type":"ContainerStarted","Data":"0233dcad00ccd80764f33cb82a7d18ea4961e4081db6d92bcd5f4b08df46c1fe"}
Jun 15 05:57:46.193960 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:46.193494    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:46.194843 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:46.194823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-5fvck" event={"ID":"312b2aac-d3e5-457a-aacc-4efbcb38f761","Type":"ContainerStarted","Data":"2fa4004b20467d60ca84bed4776107e4549507c6ad51b99a8cf6876d7b8b5ef0"}
Jun 15 05:57:46.194962 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:46.194947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:46.210082 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:46.210010    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-jmhw5" podStartSLOduration=1.194742537 podStartE2EDuration="4.209994953s" totalImagesPullingTime="3.015252416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:46.208651144 +0000 UTC m=+604.529568430" watchObservedRunningTime="2026-06-15 05:57:46.209994953 +0000 UTC m=+604.530912239"
Jun 15 05:57:46.222630 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:46.222582    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-5fvck" podStartSLOduration=1.173059926 podStartE2EDuration="4.222562728s" totalImagesPullingTime="3.049502802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:46.221867998 +0000 UTC m=+604.542785296" watchObservedRunningTime="2026-06-15 05:57:46.222562728 +0000 UTC m=+604.543480005"
Jun 15 05:57:57.200345 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:57.200252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-5fvck"
Jun 15 05:57:57.202297 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:57.202275    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-jmhw5"
Jun 15 05:57:57.982496 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:57.982455    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cw8gh"]
Jun 15 05:57:57.985217 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:57.985194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cw8gh"
Jun 15 05:57:57.987265 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:57.987240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jun 15 05:57:57.987265 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:57.987255    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6m7z5\""
Jun 15 05:57:57.993003 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:57.992970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cw8gh"]
Jun 15 05:57:58.091158 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.091116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fh6p\" (UniqueName: \"kubernetes.io/projected/72d11a8b-65a8-47a9-bc03-1bc0283f8499-kube-api-access-5fh6p\") pod \"s3-init-cw8gh\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") " pod="kserve/s3-init-cw8gh"
Jun 15 05:57:58.091352 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.091188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d11a8b-65a8-47a9-bc03-1bc0283f8499-model-cache\") pod \"s3-init-cw8gh\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") " pod="kserve/s3-init-cw8gh"
Jun 15 05:57:58.192033 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.191986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d11a8b-65a8-47a9-bc03-1bc0283f8499-model-cache\") pod \"s3-init-cw8gh\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") " pod="kserve/s3-init-cw8gh"
Jun 15 05:57:58.192239 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.192048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fh6p\" (UniqueName: \"kubernetes.io/projected/72d11a8b-65a8-47a9-bc03-1bc0283f8499-kube-api-access-5fh6p\") pod \"s3-init-cw8gh\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") " pod="kserve/s3-init-cw8gh"
Jun 15 05:57:58.192426 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.192402    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d11a8b-65a8-47a9-bc03-1bc0283f8499-model-cache\") pod \"s3-init-cw8gh\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") " pod="kserve/s3-init-cw8gh"
Jun 15 05:57:58.200524 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.200492    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fh6p\" (UniqueName: \"kubernetes.io/projected/72d11a8b-65a8-47a9-bc03-1bc0283f8499-kube-api-access-5fh6p\") pod \"s3-init-cw8gh\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") " pod="kserve/s3-init-cw8gh"
Jun 15 05:57:58.294877 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.294787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cw8gh"
Jun 15 05:57:58.423989 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:58.423813    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cw8gh"]
Jun 15 05:57:58.426505 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:57:58.426477    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d11a8b_65a8_47a9_bc03_1bc0283f8499.slice/crio-5ee56177c770729feb80b5cd82882e4a1c1dd876a1075fb36724b2d36f71fca0 WatchSource:0}: Error finding container 5ee56177c770729feb80b5cd82882e4a1c1dd876a1075fb36724b2d36f71fca0: Status 404 returned error can't find the container with id 5ee56177c770729feb80b5cd82882e4a1c1dd876a1075fb36724b2d36f71fca0
Jun 15 05:57:59.241181 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:57:59.241144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cw8gh" event={"ID":"72d11a8b-65a8-47a9-bc03-1bc0283f8499","Type":"ContainerStarted","Data":"5ee56177c770729feb80b5cd82882e4a1c1dd876a1075fb36724b2d36f71fca0"}
Jun 15 05:58:03.257840 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:03.257802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cw8gh" event={"ID":"72d11a8b-65a8-47a9-bc03-1bc0283f8499","Type":"ContainerStarted","Data":"0662590a9a8d88602ac9a7ace2798d07101bdd7a18a86d43e0c4b6213805315f"}
Jun 15 05:58:04.262984 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:04.262949    2579 generic.go:356] "Generic (PLEG): container finished" podID="72d11a8b-65a8-47a9-bc03-1bc0283f8499" containerID="0662590a9a8d88602ac9a7ace2798d07101bdd7a18a86d43e0c4b6213805315f" exitCode=0
Jun 15 05:58:04.263362 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:04.262995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cw8gh" event={"ID":"72d11a8b-65a8-47a9-bc03-1bc0283f8499","Type":"ContainerDied","Data":"0662590a9a8d88602ac9a7ace2798d07101bdd7a18a86d43e0c4b6213805315f"}
Jun 15 05:58:09.284377 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:09.284285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cw8gh" event={"ID":"72d11a8b-65a8-47a9-bc03-1bc0283f8499","Type":"ContainerStarted","Data":"c1c32692b2c6625989f06a523dbae1213199fe9ce71fbc5dee87ab85c87ce979"}
Jun 15 05:58:15.423559 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:15.423519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:58:26.589481 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:26.589414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:58:26.637743 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:26.637689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:58:26.692394 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:26.692364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:58:37.180450 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:37.180416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:58:40.772970 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:40.772932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:58:40.873552 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:58:40.873515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:59:14.505811 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:14.505705    2579 generic.go:356] "Generic (PLEG): container finished" podID="72d11a8b-65a8-47a9-bc03-1bc0283f8499" containerID="c1c32692b2c6625989f06a523dbae1213199fe9ce71fbc5dee87ab85c87ce979" exitCode=0
Jun 15 05:59:14.505811 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:14.505780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cw8gh" event={"ID":"72d11a8b-65a8-47a9-bc03-1bc0283f8499","Type":"ContainerDied","Data":"c1c32692b2c6625989f06a523dbae1213199fe9ce71fbc5dee87ab85c87ce979"}
Jun 15 05:59:15.638238 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:15.638210    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cw8gh"
Jun 15 05:59:15.656028 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:15.655999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fh6p\" (UniqueName: \"kubernetes.io/projected/72d11a8b-65a8-47a9-bc03-1bc0283f8499-kube-api-access-5fh6p\") pod \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") "
Jun 15 05:59:15.656135 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:15.656043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d11a8b-65a8-47a9-bc03-1bc0283f8499-model-cache\") pod \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\" (UID: \"72d11a8b-65a8-47a9-bc03-1bc0283f8499\") "
Jun 15 05:59:15.658537 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:15.658496    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d11a8b-65a8-47a9-bc03-1bc0283f8499-kube-api-access-5fh6p" (OuterVolumeSpecName: "kube-api-access-5fh6p") pod "72d11a8b-65a8-47a9-bc03-1bc0283f8499" (UID: "72d11a8b-65a8-47a9-bc03-1bc0283f8499"). InnerVolumeSpecName "kube-api-access-5fh6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:59:15.676020 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:15.675987    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d11a8b-65a8-47a9-bc03-1bc0283f8499-model-cache" (OuterVolumeSpecName: "model-cache") pod "72d11a8b-65a8-47a9-bc03-1bc0283f8499" (UID: "72d11a8b-65a8-47a9-bc03-1bc0283f8499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:59:15.756647 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:15.756610    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d11a8b-65a8-47a9-bc03-1bc0283f8499-model-cache\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:59:15.756647 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:15.756641    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fh6p\" (UniqueName: \"kubernetes.io/projected/72d11a8b-65a8-47a9-bc03-1bc0283f8499-kube-api-access-5fh6p\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 05:59:16.514469 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:16.514391    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cw8gh"
Jun 15 05:59:16.514606 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:16.514395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cw8gh" event={"ID":"72d11a8b-65a8-47a9-bc03-1bc0283f8499","Type":"ContainerDied","Data":"5ee56177c770729feb80b5cd82882e4a1c1dd876a1075fb36724b2d36f71fca0"}
Jun 15 05:59:16.514606 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:16.514497    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ee56177c770729feb80b5cd82882e4a1c1dd876a1075fb36724b2d36f71fca0"
Jun 15 05:59:16.867024 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:16.866986    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:59:16.959780 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:16.959721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:59:27.758955 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.758869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:59:27.860586 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.860549    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"]
Jun 15 05:59:27.860922 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.860904    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d11a8b-65a8-47a9-bc03-1bc0283f8499" containerName="s3-init"
Jun 15 05:59:27.861008 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.860925    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d11a8b-65a8-47a9-bc03-1bc0283f8499" containerName="s3-init"
Jun 15 05:59:27.861008 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.860939    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d11a8b-65a8-47a9-bc03-1bc0283f8499" containerName="download-hf-model"
Jun 15 05:59:27.861008 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.860947    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d11a8b-65a8-47a9-bc03-1bc0283f8499" containerName="download-hf-model"
Jun 15 05:59:27.861008 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.861007    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d11a8b-65a8-47a9-bc03-1bc0283f8499" containerName="s3-init"
Jun 15 05:59:27.864436 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.864415    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:27.867486 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.867458    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 05:59:27.867600 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.867542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jun 15 05:59:27.867600 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.867573    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-cb69f-kube-rbac-proxy-sar-config\""
Jun 15 05:59:27.867693 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.867540    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 05:59:27.867693 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.867643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-cb69f-predictor-serving-cert\""
Jun 15 05:59:27.868223 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.868206    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zszn8\""
Jun 15 05:59:27.868322 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.868240    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 05:59:27.883525 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.883494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"]
Jun 15 05:59:27.914360 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.914304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:59:27.943296 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.943261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:27.943457 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.943311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcgch\" (UniqueName: \"kubernetes.io/projected/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-api-access-bcgch\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:27.943457 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.943341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:27.943457 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.943370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:27.943457 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:27.943428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.044478 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.044389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.044478 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.044430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.044478 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.044466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.044771 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.044509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcgch\" (UniqueName: \"kubernetes.io/projected/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-api-access-bcgch\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.044771 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.044540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.045068 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.045032    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.045178 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.045161    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.045356 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.045335    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.047096 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.047079    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.054021 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.054002    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcgch\" (UniqueName: \"kubernetes.io/projected/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-api-access-bcgch\") pod \"isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.174911 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.174871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:28.309321 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.309293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"]
Jun 15 05:59:28.312004 ip-10-0-135-167 kubenswrapper[2579]: W0615 05:59:28.311977    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31d7dde_0e0f_40c5_b59a_802fb3c3022f.slice/crio-6bbdb174b7405ee7f0be82a1d17a6b456eec6075adb9515e26e8cd65964ef5ef WatchSource:0}: Error finding container 6bbdb174b7405ee7f0be82a1d17a6b456eec6075adb9515e26e8cd65964ef5ef: Status 404 returned error can't find the container with id 6bbdb174b7405ee7f0be82a1d17a6b456eec6075adb9515e26e8cd65964ef5ef
Jun 15 05:59:28.560294 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.560195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerStarted","Data":"899944000c02ae791c5a8e04165e5d031549e4618a325163f33e6f4c9caafa57"}
Jun 15 05:59:28.560294 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:28.560280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerStarted","Data":"6bbdb174b7405ee7f0be82a1d17a6b456eec6075adb9515e26e8cd65964ef5ef"}
Jun 15 05:59:32.575795 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:32.575750    2579 generic.go:356] "Generic (PLEG): container finished" podID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerID="899944000c02ae791c5a8e04165e5d031549e4618a325163f33e6f4c9caafa57" exitCode=0
Jun 15 05:59:32.576200 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:32.575797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerDied","Data":"899944000c02ae791c5a8e04165e5d031549e4618a325163f33e6f4c9caafa57"}
Jun 15 05:59:32.576925 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:32.576894    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:59:39.173565 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:39.173501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 05:59:46.641292 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:46.641239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerStarted","Data":"8a18e1aa92eaf4a18975d5f316f5a09cae84622f3c084399bb2aac07dff2de64"}
Jun 15 05:59:48.653053 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:48.653014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerStarted","Data":"cfe2f5275f82b8a0f0a98b6dfd57a8b89b7aeff9fec4a395a5651f7ed34b9c9a"}
Jun 15 05:59:50.664050 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:50.664016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerStarted","Data":"b0ebf87c9cc1d473e4d819e76241a326ffb304029f7b442a4abc5648b3da668b"}
Jun 15 05:59:50.664471 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:50.664226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:50.664471 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:50.664359    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:50.665621 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:50.665594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 05:59:50.684093 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:50.684041    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podStartSLOduration=2.510851863 podStartE2EDuration="23.684024577s" totalImagesPullingTime="17.507965309s" totalInitContainerRuntime="3.665207405s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:59:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:59:50.682951587 +0000 UTC m=+729.003868873" watchObservedRunningTime="2026-06-15 05:59:50.684024577 +0000 UTC m=+729.004941864"
Jun 15 05:59:51.667798 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:51.667755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:51.668281 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:51.667797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 05:59:51.668884 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:51.668860    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:59:52.671836 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:52.671791    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 05:59:52.672319 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:52.672247    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:59:57.676340 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:57.676310    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 05:59:57.676919 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:57.676890    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 05:59:57.677125 ip-10-0-135-167 kubenswrapper[2579]: I0615 05:59:57.677102    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:07.677639 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:07.677594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:00:07.678040 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:07.678008    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:17.677194 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:17.677149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:00:17.677634 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:17.677560    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:27.677548 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:27.677502    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:00:27.678059 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:27.678022    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:37.677103 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:37.677046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:00:37.677528 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:37.677396    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:47.677533 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:47.677486    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:00:47.678012 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:47.677920    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:57.677850 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:57.677751    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:00:57.678285 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:57.678261    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:59.155322 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:59.155275    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:00:59.155898 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:00:59.155647    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:09.155941 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:09.155909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 06:01:09.156357 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:09.156154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 06:01:22.878761 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.878710    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"]
Jun 15 06:01:22.879208 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.879066    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" containerID="cri-o://8a18e1aa92eaf4a18975d5f316f5a09cae84622f3c084399bb2aac07dff2de64" gracePeriod=30
Jun 15 06:01:22.879414 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.879358    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" containerID="cri-o://b0ebf87c9cc1d473e4d819e76241a326ffb304029f7b442a4abc5648b3da668b" gracePeriod=30
Jun 15 06:01:22.880105 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.879822    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy" containerID="cri-o://cfe2f5275f82b8a0f0a98b6dfd57a8b89b7aeff9fec4a395a5651f7ed34b9c9a" gracePeriod=30
Jun 15 06:01:22.925282 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.925246    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:01:22.968957 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.968922    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"]
Jun 15 06:01:22.972149 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.972134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:22.974325 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.974301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2af6c-predictor-serving-cert\""
Jun 15 06:01:22.974454 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.974306    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2af6c-kube-rbac-proxy-sar-config\""
Jun 15 06:01:22.984532 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.984041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"]
Jun 15 06:01:22.988855 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:22.988825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:01:23.002792 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.002756    2579 generic.go:356] "Generic (PLEG): container finished" podID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerID="cfe2f5275f82b8a0f0a98b6dfd57a8b89b7aeff9fec4a395a5651f7ed34b9c9a" exitCode=2
Jun 15 06:01:23.002940 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.002827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerDied","Data":"cfe2f5275f82b8a0f0a98b6dfd57a8b89b7aeff9fec4a395a5651f7ed34b9c9a"}
Jun 15 06:01:23.052153 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.052108    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"]
Jun 15 06:01:23.055418 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.055392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.057395 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.057367    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2af6c-predictor-serving-cert\""
Jun 15 06:01:23.057522 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.057413    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2af6c-kube-rbac-proxy-sar-config\""
Jun 15 06:01:23.058292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.058263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:01:23.065826 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.065801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"]
Jun 15 06:01:23.111421 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.111421 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.111647 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.111647 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtgs\" (UniqueName: \"kubernetes.io/projected/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-api-access-9mtgs\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.111647 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtxd\" (UniqueName: \"kubernetes.io/projected/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-api-access-9qtxd\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.111647 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-proxy-tls\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.111647 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.111647 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.111897 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.111897 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.111680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.152744 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.152629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:01:23.212982 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.212947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-proxy-tls\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.212982 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.212987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtgs\" (UniqueName: \"kubernetes.io/projected/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-api-access-9mtgs\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtxd\" (UniqueName: \"kubernetes.io/projected/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-api-access-9qtxd\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:01:23.213311    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-serving-cert: secret "isvc-sklearn-graph-raw-2af6c-predictor-serving-cert" not found
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:01:23.213399    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls podName:f66f5b22-5b10-4b20-9c63-d10c7e361d16 nodeName:}" failed. No retries permitted until 2026-06-15 06:01:23.713376339 +0000 UTC m=+822.034293604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls") pod "isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" (UID: "f66f5b22-5b10-4b20-9c63-d10c7e361d16") : secret "isvc-sklearn-graph-raw-2af6c-predictor-serving-cert" not found
Jun 15 06:01:23.213875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213472    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.214310 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213889    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.214310 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213958    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.214310 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213964    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.214310 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213979    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.214310 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.213993    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.215618 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.215596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-proxy-tls\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.222202 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.222178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtxd\" (UniqueName: \"kubernetes.io/projected/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-api-access-9qtxd\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.222317 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.222222    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtgs\" (UniqueName: \"kubernetes.io/projected/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-api-access-9mtgs\") pod \"isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.366333 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.366295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:23.497421 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.497391    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"]
Jun 15 06:01:23.499822 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:01:23.499787    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9429c7_456b_4fda_bb71_a3d8fd9d4bfe.slice/crio-26b6e3d18b89e6de63ac429860296442826d56acfe0bf1a9017448a0448941d4 WatchSource:0}: Error finding container 26b6e3d18b89e6de63ac429860296442826d56acfe0bf1a9017448a0448941d4: Status 404 returned error can't find the container with id 26b6e3d18b89e6de63ac429860296442826d56acfe0bf1a9017448a0448941d4
Jun 15 06:01:23.658878 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.658845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:01:23.718201 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.718113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.720564 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.720535    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls\") pod \"isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:23.882659 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:23.882620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:24.008536 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:24.008444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerStarted","Data":"83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91"}
Jun 15 06:01:24.008536 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:24.008500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerStarted","Data":"26b6e3d18b89e6de63ac429860296442826d56acfe0bf1a9017448a0448941d4"}
Jun 15 06:01:24.018232 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:24.018072    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"]
Jun 15 06:01:24.020293 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:01:24.020266    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66f5b22_5b10_4b20_9c63_d10c7e361d16.slice/crio-fd33f6d5d2fc7d838cd08168d15055b7d7bcddad25367e1ea072ba35aa64fbc5 WatchSource:0}: Error finding container fd33f6d5d2fc7d838cd08168d15055b7d7bcddad25367e1ea072ba35aa64fbc5: Status 404 returned error can't find the container with id fd33f6d5d2fc7d838cd08168d15055b7d7bcddad25367e1ea072ba35aa64fbc5
Jun 15 06:01:25.013563 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:25.013523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerStarted","Data":"423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509"}
Jun 15 06:01:25.013981 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:25.013599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerStarted","Data":"fd33f6d5d2fc7d838cd08168d15055b7d7bcddad25367e1ea072ba35aa64fbc5"}
Jun 15 06:01:27.672994 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:27.672948    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jun 15 06:01:28.026550 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:28.026511    2579 generic.go:356] "Generic (PLEG): container finished" podID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerID="423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509" exitCode=0
Jun 15 06:01:28.026711 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:28.026582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerDied","Data":"423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509"}
Jun 15 06:01:28.029088 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:28.029063    2579 generic.go:356] "Generic (PLEG): container finished" podID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerID="8a18e1aa92eaf4a18975d5f316f5a09cae84622f3c084399bb2aac07dff2de64" exitCode=0
Jun 15 06:01:28.029203 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:28.029152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerDied","Data":"8a18e1aa92eaf4a18975d5f316f5a09cae84622f3c084399bb2aac07dff2de64"}
Jun 15 06:01:28.030632 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:28.030609    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerID="83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91" exitCode=0
Jun 15 06:01:28.030752 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:28.030668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerDied","Data":"83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91"}
Jun 15 06:01:29.038624 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.038591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerStarted","Data":"db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723"}
Jun 15 06:01:29.039059 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.038659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerStarted","Data":"315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557"}
Jun 15 06:01:29.039125 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.039072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:29.039190 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.039136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:29.040719 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.040692    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:01:29.059438 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.059376    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podStartSLOduration=3.145791981 podStartE2EDuration="7.059353847s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.913561866s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:01:29.056632713 +0000 UTC m=+827.377549999" watchObservedRunningTime="2026-06-15 06:01:29.059353847 +0000 UTC m=+827.380271135"
Jun 15 06:01:29.155709 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.155242    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:01:29.155709 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:29.155643    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:30.043940 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:30.043535    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:01:32.672785 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:32.672715    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jun 15 06:01:35.048697 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:35.048666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:01:35.049296 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:35.049266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:01:37.672043 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:37.672000    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jun 15 06:01:37.672452 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:37.672139    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 06:01:39.155845 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:39.155795    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:01:39.156280 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:39.156180    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:42.673161 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:42.673120    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jun 15 06:01:45.050234 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:45.050191    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:01:47.672494 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:47.672450    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jun 15 06:01:49.155430 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:49.155306    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jun 15 06:01:49.155926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:49.155449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 06:01:49.155926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:49.155655    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:49.155926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:49.155838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 06:01:50.135559 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:50.135523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerStarted","Data":"069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048"}
Jun 15 06:01:50.135755 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:50.135625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerStarted","Data":"f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4"}
Jun 15 06:01:50.135846 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:50.135828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:50.152966 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:50.152907    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podStartSLOduration=1.806537811 podStartE2EDuration="27.15288452s" totalImagesPullingTime="21.445198558s" totalInitContainerRuntime="3.901148151s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:01:50.151493678 +0000 UTC m=+848.472410965" watchObservedRunningTime="2026-06-15 06:01:50.15288452 +0000 UTC m=+848.473801807"
Jun 15 06:01:51.139160 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:51.139129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:51.140284 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:51.140257    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:01:52.142853 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:52.142815    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:01:52.672633 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:52.672586    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jun 15 06:01:53.149240 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.149206    2579 generic.go:356] "Generic (PLEG): container finished" podID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerID="b0ebf87c9cc1d473e4d819e76241a326ffb304029f7b442a4abc5648b3da668b" exitCode=0
Jun 15 06:01:53.149623 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.149289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerDied","Data":"b0ebf87c9cc1d473e4d819e76241a326ffb304029f7b442a4abc5648b3da668b"}
Jun 15 06:01:53.584243 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.584217    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 06:01:53.680172 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-rbac-proxy-sar-config\") pod \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") "
Jun 15 06:01:53.680374 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kserve-provision-location\") pod \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") "
Jun 15 06:01:53.680374 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680224    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcgch\" (UniqueName: \"kubernetes.io/projected/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-api-access-bcgch\") pod \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") "
Jun 15 06:01:53.680374 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-cabundle-cert\") pod \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") "
Jun 15 06:01:53.680374 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-proxy-tls\") pod \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\" (UID: \"c31d7dde-0e0f-40c5-b59a-802fb3c3022f\") "
Jun 15 06:01:53.680641 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680619    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c31d7dde-0e0f-40c5-b59a-802fb3c3022f" (UID: "c31d7dde-0e0f-40c5-b59a-802fb3c3022f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:01:53.680687 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680635    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c31d7dde-0e0f-40c5-b59a-802fb3c3022f" (UID: "c31d7dde-0e0f-40c5-b59a-802fb3c3022f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:01:53.680687 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.680616    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c31d7dde-0e0f-40c5-b59a-802fb3c3022f" (UID: "c31d7dde-0e0f-40c5-b59a-802fb3c3022f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:01:53.682521 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.682496    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-api-access-bcgch" (OuterVolumeSpecName: "kube-api-access-bcgch") pod "c31d7dde-0e0f-40c5-b59a-802fb3c3022f" (UID: "c31d7dde-0e0f-40c5-b59a-802fb3c3022f"). InnerVolumeSpecName "kube-api-access-bcgch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:01:53.682745 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.682691    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c31d7dde-0e0f-40c5-b59a-802fb3c3022f" (UID: "c31d7dde-0e0f-40c5-b59a-802fb3c3022f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:01:53.781279 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.781234    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:01:53.781279 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.781272    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcgch\" (UniqueName: \"kubernetes.io/projected/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-api-access-bcgch\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:01:53.781279 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.781283    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:01:53.781279 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.781292    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:01:53.781548 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:53.781300    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c31d7dde-0e0f-40c5-b59a-802fb3c3022f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:01:54.155637 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.155608    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"
Jun 15 06:01:54.158396 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.158365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7" event={"ID":"c31d7dde-0e0f-40c5-b59a-802fb3c3022f","Type":"ContainerDied","Data":"6bbdb174b7405ee7f0be82a1d17a6b456eec6075adb9515e26e8cd65964ef5ef"}
Jun 15 06:01:54.158549 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.158420    2579 scope.go:117] "RemoveContainer" containerID="b0ebf87c9cc1d473e4d819e76241a326ffb304029f7b442a4abc5648b3da668b"
Jun 15 06:01:54.167007 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.166988    2579 scope.go:117] "RemoveContainer" containerID="cfe2f5275f82b8a0f0a98b6dfd57a8b89b7aeff9fec4a395a5651f7ed34b9c9a"
Jun 15 06:01:54.174779 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.174747    2579 scope.go:117] "RemoveContainer" containerID="8a18e1aa92eaf4a18975d5f316f5a09cae84622f3c084399bb2aac07dff2de64"
Jun 15 06:01:54.180114 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.180083    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"]
Jun 15 06:01:54.183205 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.183189    2579 scope.go:117] "RemoveContainer" containerID="899944000c02ae791c5a8e04165e5d031549e4618a325163f33e6f4c9caafa57"
Jun 15 06:01:54.184215 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:54.184141    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-cb69f-predictor-5b69cd895c-tcpf7"]
Jun 15 06:01:55.050169 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:55.050123    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:01:56.159461 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:56.159428    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" path="/var/lib/kubelet/pods/c31d7dde-0e0f-40c5-b59a-802fb3c3022f/volumes"
Jun 15 06:01:57.147248 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:57.147218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:01:57.147690 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:01:57.147652    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:02:05.049335 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:05.049295    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:02:07.147831 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:07.147785    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:02:15.049901 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:15.049850    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:02:17.147869 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:17.147831    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:02:25.049318 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:25.049278    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:02:27.148511 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:27.148427    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:02:35.049744 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:35.049693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:02:37.148206 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:37.148168    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:02:42.154770 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:42.154741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 06:02:42.155439 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:42.155414    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 06:02:45.049747 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:45.049687    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:02:47.148375 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:47.148331    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:02:55.050516 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:55.050485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:02:57.148885 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:02:57.148856    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:03:13.045918 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:13.045883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:03:13.404468 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:13.404430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:03:23.151889 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.151845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:03:23.258991 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.258958    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"]
Jun 15 06:03:23.259237 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.259213    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" containerID="cri-o://315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557" gracePeriod=30
Jun 15 06:03:23.259307 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.259257    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kube-rbac-proxy" containerID="cri-o://db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723" gracePeriod=30
Jun 15 06:03:23.314998 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.314963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"]
Jun 15 06:03:23.315249 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315237    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="storage-initializer"
Jun 15 06:03:23.315292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315251    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="storage-initializer"
Jun 15 06:03:23.315292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315260    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container"
Jun 15 06:03:23.315292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315266    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container"
Jun 15 06:03:23.315292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315279    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy"
Jun 15 06:03:23.315292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315284    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy"
Jun 15 06:03:23.315292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315293    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent"
Jun 15 06:03:23.315572 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315298    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent"
Jun 15 06:03:23.315572 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315343    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kube-rbac-proxy"
Jun 15 06:03:23.315572 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315352    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="kserve-container"
Jun 15 06:03:23.315572 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.315359    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31d7dde-0e0f-40c5-b59a-802fb3c3022f" containerName="agent"
Jun 15 06:03:23.318853 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.318829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.321071 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.321048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-ea1cf-kube-rbac-proxy-sar-config\""
Jun 15 06:03:23.321186 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.321107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-serving-cert\""
Jun 15 06:03:23.329209 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.329182    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"]
Jun 15 06:03:23.350492 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.350462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.350684 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.350509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754k7\" (UniqueName: \"kubernetes.io/projected/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-api-access-754k7\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.350684 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.350592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.350684 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.350641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.350684 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.350678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.447471 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.447387    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"]
Jun 15 06:03:23.447749 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.447692    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" containerID="cri-o://f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4" gracePeriod=30
Jun 15 06:03:23.447922 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.447774    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kube-rbac-proxy" containerID="cri-o://069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048" gracePeriod=30
Jun 15 06:03:23.451619 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.451591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754k7\" (UniqueName: \"kubernetes.io/projected/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-api-access-754k7\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.451796 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.451643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.451796 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.451673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.451796 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.451700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.451796 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.451773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.452221 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.452192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.452526 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.452496    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.452632 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.452547    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.454549 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.454529    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.459203 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.459180    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"]
Jun 15 06:03:23.463007 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.462990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.463866 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.463847    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-754k7\" (UniqueName: \"kubernetes.io/projected/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-api-access-754k7\") pod \"isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.465386 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.465362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-serving-cert\""
Jun 15 06:03:23.465478 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.465363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-ea1cf-kube-rbac-proxy-sar-config\""
Jun 15 06:03:23.470481 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.470460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"]
Jun 15 06:03:23.486175 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.486146    2579 generic.go:356] "Generic (PLEG): container finished" podID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerID="db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723" exitCode=2
Jun 15 06:03:23.486289 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.486224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerDied","Data":"db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723"}
Jun 15 06:03:23.553100 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.553052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.553291 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.553121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.553291 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.553165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.553291 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.553275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb57b\" (UniqueName: \"kubernetes.io/projected/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-api-access-qb57b\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.553476 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.553326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.632089 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.632049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:23.654335 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.654303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.654486 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.654356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb57b\" (UniqueName: \"kubernetes.io/projected/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-api-access-qb57b\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.654486 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.654381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.654486 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.654451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.654615 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.654540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.654919 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.654891    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.655199 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.655183    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.655344 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.655323    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.656874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.656856    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.663194 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.663173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb57b\" (UniqueName: \"kubernetes.io/projected/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-api-access-qb57b\") pod \"isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.761939 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.761912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"]
Jun 15 06:03:23.764355 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:03:23.764330    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c43bf0_8ba9_49d7_8dc9_af68f3a2c07d.slice/crio-2a720493f0227ae241ed17e815a327d6d77ce6859f6218c00c99ce19b0147a13 WatchSource:0}: Error finding container 2a720493f0227ae241ed17e815a327d6d77ce6859f6218c00c99ce19b0147a13: Status 404 returned error can't find the container with id 2a720493f0227ae241ed17e815a327d6d77ce6859f6218c00c99ce19b0147a13
Jun 15 06:03:23.784656 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.784630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:23.928813 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:23.928765    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"]
Jun 15 06:03:23.931258 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:03:23.931211    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeb06b4_e61d_4d7b_89a3_5095f6ac6ebf.slice/crio-6727e17e3f665a0ff28ad9d2f0fdf3eee18eca9fc3423f8593c679d34c579f17 WatchSource:0}: Error finding container 6727e17e3f665a0ff28ad9d2f0fdf3eee18eca9fc3423f8593c679d34c579f17: Status 404 returned error can't find the container with id 6727e17e3f665a0ff28ad9d2f0fdf3eee18eca9fc3423f8593c679d34c579f17
Jun 15 06:03:24.099012 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:24.098970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:03:24.493488 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:24.493448    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerID="069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048" exitCode=2
Jun 15 06:03:24.493957 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:24.493518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerDied","Data":"069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048"}
Jun 15 06:03:24.495301 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:24.495264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerStarted","Data":"e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac"}
Jun 15 06:03:24.495434 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:24.495335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerStarted","Data":"6727e17e3f665a0ff28ad9d2f0fdf3eee18eca9fc3423f8593c679d34c579f17"}
Jun 15 06:03:24.496638 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:24.496617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerStarted","Data":"ff2064b7f395788a0bafbbadc36250b64401bfe3d34d05d8fb2c8fa712bd8345"}
Jun 15 06:03:24.496768 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:24.496670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerStarted","Data":"2a720493f0227ae241ed17e815a327d6d77ce6859f6218c00c99ce19b0147a13"}
Jun 15 06:03:25.044452 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:25.044410    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jun 15 06:03:25.050074 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:25.050048    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jun 15 06:03:27.143194 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.143149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jun 15 06:03:27.147798 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.147769    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jun 15 06:03:27.388768 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.388742    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:03:27.481498 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.481399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-rbac-proxy-sar-config\") pod \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") "
Jun 15 06:03:27.481498 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.481432    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-cabundle-cert\") pod \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") "
Jun 15 06:03:27.481498 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.481461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-proxy-tls\") pod \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") "
Jun 15 06:03:27.481498 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.481478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtgs\" (UniqueName: \"kubernetes.io/projected/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-api-access-9mtgs\") pod \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") "
Jun 15 06:03:27.481874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.481594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kserve-provision-location\") pod \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\" (UID: \"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe\") "
Jun 15 06:03:27.481936 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.481904    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" (UID: "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:03:27.481991 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.481970    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" (UID: "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:03:27.482048 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.482026    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" (UID: "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:03:27.483604 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.483581    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" (UID: "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:03:27.483716 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.483699    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-api-access-9mtgs" (OuterVolumeSpecName: "kube-api-access-9mtgs") pod "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" (UID: "8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe"). InnerVolumeSpecName "kube-api-access-9mtgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:03:27.508897 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.508861    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerID="f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4" exitCode=0
Jun 15 06:03:27.509047 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.508930    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"
Jun 15 06:03:27.509047 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.508947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerDied","Data":"f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4"}
Jun 15 06:03:27.509047 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.509009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh" event={"ID":"8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe","Type":"ContainerDied","Data":"26b6e3d18b89e6de63ac429860296442826d56acfe0bf1a9017448a0448941d4"}
Jun 15 06:03:27.509047 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.509038    2579 scope.go:117] "RemoveContainer" containerID="069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048"
Jun 15 06:03:27.520821 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.520796    2579 scope.go:117] "RemoveContainer" containerID="f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4"
Jun 15 06:03:27.529489 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.529467    2579 scope.go:117] "RemoveContainer" containerID="83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91"
Jun 15 06:03:27.532966 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.532939    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"]
Jun 15 06:03:27.536580 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.536557    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2af6c-predictor-75f5969657-v4bqh"]
Jun 15 06:03:27.537807 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.537786    2579 scope.go:117] "RemoveContainer" containerID="069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048"
Jun 15 06:03:27.538106 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:03:27.538087    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048\": container with ID starting with 069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048 not found: ID does not exist" containerID="069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048"
Jun 15 06:03:27.538155 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.538115    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048"} err="failed to get container status \"069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048\": rpc error: code = NotFound desc = could not find container \"069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048\": container with ID starting with 069e56b89c2201d111a70f2fe5c8e07a43b7bc4b12def9f8e52ca2a32c563048 not found: ID does not exist"
Jun 15 06:03:27.538155 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.538133    2579 scope.go:117] "RemoveContainer" containerID="f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4"
Jun 15 06:03:27.538389 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:03:27.538373    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4\": container with ID starting with f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4 not found: ID does not exist" containerID="f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4"
Jun 15 06:03:27.538441 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.538396    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4"} err="failed to get container status \"f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4\": rpc error: code = NotFound desc = could not find container \"f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4\": container with ID starting with f1151795d92e11b3a5fbceac61cfc7041a37ab4c8301cdb08c1d6b70a67403d4 not found: ID does not exist"
Jun 15 06:03:27.538441 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.538409    2579 scope.go:117] "RemoveContainer" containerID="83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91"
Jun 15 06:03:27.538626 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:03:27.538609    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91\": container with ID starting with 83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91 not found: ID does not exist" containerID="83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91"
Jun 15 06:03:27.538666 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.538631    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91"} err="failed to get container status \"83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91\": rpc error: code = NotFound desc = could not find container \"83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91\": container with ID starting with 83d45e143a9086551416b5765e7c8a8cac5ae9dd424b35fbcaf89483aa03ef91 not found: ID does not exist"
Jun 15 06:03:27.582476 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.582441    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:27.582476 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.582472    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:27.582476 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.582484    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:27.582696 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.582493    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtgs\" (UniqueName: \"kubernetes.io/projected/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kube-api-access-9mtgs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:27.582696 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:27.582502    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:28.159409 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.159373    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" path="/var/lib/kubelet/pods/8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe/volumes"
Jun 15 06:03:28.311697 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.311671    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:03:28.389953 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.389912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kserve-provision-location\") pod \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") "
Jun 15 06:03:28.389953 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.389955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-cabundle-cert\") pod \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") "
Jun 15 06:03:28.390171 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.389997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-rbac-proxy-sar-config\") pod \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") "
Jun 15 06:03:28.390171 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.390027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls\") pod \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") "
Jun 15 06:03:28.390171 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.390063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qtxd\" (UniqueName: \"kubernetes.io/projected/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-api-access-9qtxd\") pod \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\" (UID: \"f66f5b22-5b10-4b20-9c63-d10c7e361d16\") "
Jun 15 06:03:28.390331 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.390313    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f66f5b22-5b10-4b20-9c63-d10c7e361d16" (UID: "f66f5b22-5b10-4b20-9c63-d10c7e361d16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:03:28.390460 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.390434    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f66f5b22-5b10-4b20-9c63-d10c7e361d16" (UID: "f66f5b22-5b10-4b20-9c63-d10c7e361d16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:03:28.390495 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.390442    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f66f5b22-5b10-4b20-9c63-d10c7e361d16" (UID: "f66f5b22-5b10-4b20-9c63-d10c7e361d16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:03:28.392311 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.392288    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f66f5b22-5b10-4b20-9c63-d10c7e361d16" (UID: "f66f5b22-5b10-4b20-9c63-d10c7e361d16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:03:28.392371 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.392338    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-api-access-9qtxd" (OuterVolumeSpecName: "kube-api-access-9qtxd") pod "f66f5b22-5b10-4b20-9c63-d10c7e361d16" (UID: "f66f5b22-5b10-4b20-9c63-d10c7e361d16"). InnerVolumeSpecName "kube-api-access-9qtxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:03:28.490577 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.490531    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:28.490577 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.490564    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:28.490577 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.490579    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:28.490577 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.490591    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f66f5b22-5b10-4b20-9c63-d10c7e361d16-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:28.490882 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.490602    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtxd\" (UniqueName: \"kubernetes.io/projected/f66f5b22-5b10-4b20-9c63-d10c7e361d16-kube-api-access-9qtxd\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:03:28.514515 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.514484    2579 generic.go:356] "Generic (PLEG): container finished" podID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerID="ff2064b7f395788a0bafbbadc36250b64401bfe3d34d05d8fb2c8fa712bd8345" exitCode=0
Jun 15 06:03:28.514641 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.514556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerDied","Data":"ff2064b7f395788a0bafbbadc36250b64401bfe3d34d05d8fb2c8fa712bd8345"}
Jun 15 06:03:28.517370 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.517345    2579 generic.go:356] "Generic (PLEG): container finished" podID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerID="315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557" exitCode=0
Jun 15 06:03:28.517473 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.517423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerDied","Data":"315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557"}
Jun 15 06:03:28.517529 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.517431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"
Jun 15 06:03:28.517529 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.517496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f" event={"ID":"f66f5b22-5b10-4b20-9c63-d10c7e361d16","Type":"ContainerDied","Data":"fd33f6d5d2fc7d838cd08168d15055b7d7bcddad25367e1ea072ba35aa64fbc5"}
Jun 15 06:03:28.517625 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.517532    2579 scope.go:117] "RemoveContainer" containerID="db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723"
Jun 15 06:03:28.519138 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.519105    2579 generic.go:356] "Generic (PLEG): container finished" podID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerID="e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac" exitCode=0
Jun 15 06:03:28.519248 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.519154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerDied","Data":"e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac"}
Jun 15 06:03:28.528207 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.528183    2579 scope.go:117] "RemoveContainer" containerID="315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557"
Jun 15 06:03:28.537772 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.537581    2579 scope.go:117] "RemoveContainer" containerID="423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509"
Jun 15 06:03:28.554944 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.554921    2579 scope.go:117] "RemoveContainer" containerID="db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723"
Jun 15 06:03:28.555342 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:03:28.555319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723\": container with ID starting with db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723 not found: ID does not exist" containerID="db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723"
Jun 15 06:03:28.555427 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.555351    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723"} err="failed to get container status \"db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723\": rpc error: code = NotFound desc = could not find container \"db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723\": container with ID starting with db9ef56b8fb413f1958880ceeef03e24aa4aff50575d0b02e26df70c93e97723 not found: ID does not exist"
Jun 15 06:03:28.555427 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.555369    2579 scope.go:117] "RemoveContainer" containerID="315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557"
Jun 15 06:03:28.555673 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:03:28.555654    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557\": container with ID starting with 315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557 not found: ID does not exist" containerID="315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557"
Jun 15 06:03:28.555718 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.555681    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557"} err="failed to get container status \"315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557\": rpc error: code = NotFound desc = could not find container \"315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557\": container with ID starting with 315da0cfd4bc0934fb18be589ea36e5606b3d067bd03b5c865be9fd24265e557 not found: ID does not exist"
Jun 15 06:03:28.555718 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.555697    2579 scope.go:117] "RemoveContainer" containerID="423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509"
Jun 15 06:03:28.555996 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:03:28.555976    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509\": container with ID starting with 423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509 not found: ID does not exist" containerID="423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509"
Jun 15 06:03:28.556041 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.556002    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509"} err="failed to get container status \"423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509\": rpc error: code = NotFound desc = could not find container \"423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509\": container with ID starting with 423a9b53cb82e09123993886f3e98e36e249ce54536c9aaa5b8ffbce91f8c509 not found: ID does not exist"
Jun 15 06:03:28.560514 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.560492    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"]
Jun 15 06:03:28.564033 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:28.564000    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2af6c-predictor-74db776545-ldr9f"]
Jun 15 06:03:29.526650 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.526604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerStarted","Data":"cfd8b9a2ed5746092d8c22b6389b00bc6bedb2c43862705b1914253b1ad19b81"}
Jun 15 06:03:29.527130 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.526703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerStarted","Data":"13a059c580f2e800c282161fc314167ce7e7fb399e0916408ab25c8a0ed8e6e2"}
Jun 15 06:03:29.527130 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.526991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:29.529921 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.529895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerStarted","Data":"c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9"}
Jun 15 06:03:29.530067 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.529957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerStarted","Data":"cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a"}
Jun 15 06:03:29.530267 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.530246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:29.530398 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.530295    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:29.531660 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.531635    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:03:29.545912 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.545854    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podStartSLOduration=2.79176236 podStartE2EDuration="6.545839042s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754076682s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:03:29.543900849 +0000 UTC m=+947.864818136" watchObservedRunningTime="2026-06-15 06:03:29.545839042 +0000 UTC m=+947.866756327"
Jun 15 06:03:29.560682 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:29.560629    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podStartSLOduration=2.550420142 podStartE2EDuration="6.560613708s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.010193566s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:03:29.559355219 +0000 UTC m=+947.880272505" watchObservedRunningTime="2026-06-15 06:03:29.560613708 +0000 UTC m=+947.881530993"
Jun 15 06:03:30.159709 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:30.159674    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" path="/var/lib/kubelet/pods/f66f5b22-5b10-4b20-9c63-d10c7e361d16/volumes"
Jun 15 06:03:30.534471 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:30.534364    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:03:30.534884 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:30.534381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:30.535458 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:30.535436    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:03:31.537783 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:31.537743    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:03:35.539386 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:35.539358    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:03:35.539959 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:35.539925    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:03:36.543133 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:36.543108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:03:36.543637 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:36.543605    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:03:45.540431 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:45.540392    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:03:46.544040 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:46.544001    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:03:55.540322 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:55.540279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:03:56.544084 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:03:56.543993    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:04:05.540531 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:05.540490    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:04:06.543634 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:06.543595    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:04:15.540558 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:15.540520    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:04:16.543741 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:16.543683    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:04:25.540513 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:25.540469    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:04:26.543736 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:26.543680    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:04:26.713290 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:26.713247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:04:26.811917 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:26.811831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:04:30.860584 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:30.860546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:04:30.907603 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:30.907559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:04:35.540488 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:35.540456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:04:36.544025 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:36.543989    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:04:37.618436 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:37.618391    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:04:46.545029 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:04:46.544998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:05:03.357917 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:03.357883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:03.807994 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:03.807884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:13.431900 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.431860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:13.521880 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.521849    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"]
Jun 15 06:05:13.522145 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.522119    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" containerID="cri-o://cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a" gracePeriod=30
Jun 15 06:05:13.522230 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.522119    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kube-rbac-proxy" containerID="cri-o://c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9" gracePeriod=30
Jun 15 06:05:13.551888 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.551856    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:13.589421 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589377    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"]
Jun 15 06:05:13.589766 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589719    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kube-rbac-proxy"
Jun 15 06:05:13.589766 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589768    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kube-rbac-proxy"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589791    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="storage-initializer"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589800    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="storage-initializer"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589824    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="storage-initializer"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589832    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="storage-initializer"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589842    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589850    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589862    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589870    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589882    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kube-rbac-proxy"
Jun 15 06:05:13.589926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589891    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kube-rbac-proxy"
Jun 15 06:05:13.590377 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589970    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kserve-container"
Jun 15 06:05:13.590377 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589982    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f66f5b22-5b10-4b20-9c63-d10c7e361d16" containerName="kube-rbac-proxy"
Jun 15 06:05:13.590377 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.589992    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kserve-container"
Jun 15 06:05:13.590377 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.590005    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e9429c7-456b-4fda-bb71-a3d8fd9d4bfe" containerName="kube-rbac-proxy"
Jun 15 06:05:13.593159 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.593139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.595610 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.595461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-e564c-kube-rbac-proxy-sar-config\""
Jun 15 06:05:13.595610 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.595474    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-e564c-predictor-serving-cert\""
Jun 15 06:05:13.604255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.604230    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"]
Jun 15 06:05:13.631103 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.631066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:13.645794 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.644758    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"]
Jun 15 06:05:13.645794 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.645172    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" containerID="cri-o://13a059c580f2e800c282161fc314167ce7e7fb399e0916408ab25c8a0ed8e6e2" gracePeriod=30
Jun 15 06:05:13.646514 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.646471    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kube-rbac-proxy" containerID="cri-o://cfd8b9a2ed5746092d8c22b6389b00bc6bedb2c43862705b1914253b1ad19b81" gracePeriod=30
Jun 15 06:05:13.681262 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.681229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e18deea-398e-4c1a-9f4b-900af722df84-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.681441 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.681279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x582m\" (UniqueName: \"kubernetes.io/projected/9e18deea-398e-4c1a-9f4b-900af722df84-kube-api-access-x582m\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.681441 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.681357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.725248 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.725156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:13.782779 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.782710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x582m\" (UniqueName: \"kubernetes.io/projected/9e18deea-398e-4c1a-9f4b-900af722df84-kube-api-access-x582m\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.782987 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.782891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.782987 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.782951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e18deea-398e-4c1a-9f4b-900af722df84-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.783095 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:05:13.783012    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-serving-cert: secret "message-dumper-raw-e564c-predictor-serving-cert" not found
Jun 15 06:05:13.783095 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:05:13.783084    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls podName:9e18deea-398e-4c1a-9f4b-900af722df84 nodeName:}" failed. No retries permitted until 2026-06-15 06:05:14.283064999 +0000 UTC m=+1052.603982265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls") pod "message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" (UID: "9e18deea-398e-4c1a-9f4b-900af722df84") : secret "message-dumper-raw-e564c-predictor-serving-cert" not found
Jun 15 06:05:13.783531 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.783513    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e18deea-398e-4c1a-9f4b-900af722df84-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.793312 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.793284    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x582m\" (UniqueName: \"kubernetes.io/projected/9e18deea-398e-4c1a-9f4b-900af722df84-kube-api-access-x582m\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:13.914161 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.914126    2579 generic.go:356] "Generic (PLEG): container finished" podID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerID="cfd8b9a2ed5746092d8c22b6389b00bc6bedb2c43862705b1914253b1ad19b81" exitCode=2
Jun 15 06:05:13.914325 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.914201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerDied","Data":"cfd8b9a2ed5746092d8c22b6389b00bc6bedb2c43862705b1914253b1ad19b81"}
Jun 15 06:05:13.916267 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.916244    2579 generic.go:356] "Generic (PLEG): container finished" podID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerID="c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9" exitCode=2
Jun 15 06:05:13.916542 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.916283    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerDied","Data":"c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9"}
Jun 15 06:05:13.924309 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:13.924260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:14.286418 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:14.286377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:14.288823 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:14.288789    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls\") pod \"message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") " pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:14.504504 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:14.504460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:14.633166 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:14.633047    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"]
Jun 15 06:05:14.635393 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:05:14.635364    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e18deea_398e_4c1a_9f4b_900af722df84.slice/crio-6590dad55e38777339e94f188371442dd27b2e2f7b2841186790825480f9df18 WatchSource:0}: Error finding container 6590dad55e38777339e94f188371442dd27b2e2f7b2841186790825480f9df18: Status 404 returned error can't find the container with id 6590dad55e38777339e94f188371442dd27b2e2f7b2841186790825480f9df18
Jun 15 06:05:14.637470 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:14.637455    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:05:14.921439 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:14.921407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" event={"ID":"9e18deea-398e-4c1a-9f4b-900af722df84","Type":"ContainerStarted","Data":"6590dad55e38777339e94f188371442dd27b2e2f7b2841186790825480f9df18"}
Jun 15 06:05:15.534539 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:15.534499    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jun 15 06:05:15.540028 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:15.539989    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jun 15 06:05:16.538716 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:16.538674    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jun 15 06:05:16.543627 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:16.543596    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jun 15 06:05:16.933971 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:16.933934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" event={"ID":"9e18deea-398e-4c1a-9f4b-900af722df84","Type":"ContainerStarted","Data":"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"}
Jun 15 06:05:16.934132 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:16.934004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" event={"ID":"9e18deea-398e-4c1a-9f4b-900af722df84","Type":"ContainerStarted","Data":"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"}
Jun 15 06:05:16.934132 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:16.934036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:16.950909 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:16.950858    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" podStartSLOduration=2.639716355 podStartE2EDuration="3.950841183s" totalImagesPullingTime="1.311124828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:05:16.949715719 +0000 UTC m=+1055.270633004" watchObservedRunningTime="2026-06-15 06:05:16.950841183 +0000 UTC m=+1055.271758469"
Jun 15 06:05:17.771912 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.771885    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:05:17.815719 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.815686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-proxy-tls\") pod \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") "
Jun 15 06:05:17.815920 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.815755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-rbac-proxy-sar-config\") pod \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") "
Jun 15 06:05:17.815920 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.815787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb57b\" (UniqueName: \"kubernetes.io/projected/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-api-access-qb57b\") pod \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") "
Jun 15 06:05:17.815920 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.815833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kserve-provision-location\") pod \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") "
Jun 15 06:05:17.815920 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.815875    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-cabundle-cert\") pod \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\" (UID: \"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf\") "
Jun 15 06:05:17.816149 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.816107    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" (UID: "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:05:17.816218 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.816157    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" (UID: "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:05:17.816363 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.816338    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" (UID: "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:05:17.817930 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.817913    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" (UID: "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:05:17.818089 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.818069    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-api-access-qb57b" (OuterVolumeSpecName: "kube-api-access-qb57b") pod "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" (UID: "cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf"). InnerVolumeSpecName "kube-api-access-qb57b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:05:17.916667 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.916566    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:17.916667 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.916602    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:17.916667 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.916612    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb57b\" (UniqueName: \"kubernetes.io/projected/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kube-api-access-qb57b\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:17.916667 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.916621    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:17.916667 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.916631    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:17.940139 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.940107    2579 generic.go:356] "Generic (PLEG): container finished" podID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerID="cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a" exitCode=0
Jun 15 06:05:17.940289 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.940152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerDied","Data":"cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a"}
Jun 15 06:05:17.940289 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.940214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq" event={"ID":"cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf","Type":"ContainerDied","Data":"6727e17e3f665a0ff28ad9d2f0fdf3eee18eca9fc3423f8593c679d34c579f17"}
Jun 15 06:05:17.940289 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.940167    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"
Jun 15 06:05:17.940289 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.940249    2579 scope.go:117] "RemoveContainer" containerID="c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9"
Jun 15 06:05:17.940706 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.940645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:17.942613 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.942593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:17.952720 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.949065    2579 scope.go:117] "RemoveContainer" containerID="cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a"
Jun 15 06:05:17.960202 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.960178    2579 scope.go:117] "RemoveContainer" containerID="e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac"
Jun 15 06:05:17.967709 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.967687    2579 scope.go:117] "RemoveContainer" containerID="c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9"
Jun 15 06:05:17.967993 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:05:17.967972    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9\": container with ID starting with c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9 not found: ID does not exist" containerID="c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9"
Jun 15 06:05:17.968057 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.968002    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9"} err="failed to get container status \"c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9\": rpc error: code = NotFound desc = could not find container \"c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9\": container with ID starting with c02c96cde6f6a32a04277dc5073993bc9bc11775076ec4f2f6bc59fb7f9eccf9 not found: ID does not exist"
Jun 15 06:05:17.968057 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.968021    2579 scope.go:117] "RemoveContainer" containerID="cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a"
Jun 15 06:05:17.968239 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:05:17.968224    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a\": container with ID starting with cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a not found: ID does not exist" containerID="cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a"
Jun 15 06:05:17.968280 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.968243    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a"} err="failed to get container status \"cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a\": rpc error: code = NotFound desc = could not find container \"cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a\": container with ID starting with cde94d2bda621bc5100493c86700ab8b7563b3a6a4d0f899be05355d7427475a not found: ID does not exist"
Jun 15 06:05:17.968280 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.968256    2579 scope.go:117] "RemoveContainer" containerID="e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac"
Jun 15 06:05:17.968496 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:05:17.968479    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac\": container with ID starting with e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac not found: ID does not exist" containerID="e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac"
Jun 15 06:05:17.968535 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.968501    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac"} err="failed to get container status \"e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac\": rpc error: code = NotFound desc = could not find container \"e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac\": container with ID starting with e388520c974f6f43e090297a4cf84f933382f39b4745da5ff800370813b64dac not found: ID does not exist"
Jun 15 06:05:17.975356 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.975331    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"]
Jun 15 06:05:17.979189 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:17.979160    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-ea1cf-predictor-97f4566d9-nf7sq"]
Jun 15 06:05:18.158788 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:18.158751    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" path="/var/lib/kubelet/pods/cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf/volumes"
Jun 15 06:05:18.946532 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:18.946504    2579 generic.go:356] "Generic (PLEG): container finished" podID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerID="13a059c580f2e800c282161fc314167ce7e7fb399e0916408ab25c8a0ed8e6e2" exitCode=0
Jun 15 06:05:18.946940 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:18.946577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerDied","Data":"13a059c580f2e800c282161fc314167ce7e7fb399e0916408ab25c8a0ed8e6e2"}
Jun 15 06:05:18.998774 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:18.998744    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:05:19.125522 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.125485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-proxy-tls\") pod \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") "
Jun 15 06:05:19.125680 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.125539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-rbac-proxy-sar-config\") pod \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") "
Jun 15 06:05:19.125680 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.125655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-754k7\" (UniqueName: \"kubernetes.io/projected/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-api-access-754k7\") pod \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") "
Jun 15 06:05:19.125828 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.125689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-cabundle-cert\") pod \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") "
Jun 15 06:05:19.125828 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.125713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kserve-provision-location\") pod \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\" (UID: \"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d\") "
Jun 15 06:05:19.126069 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.126036    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" (UID: "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:05:19.126198 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.126075    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" (UID: "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:05:19.126198 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.126089    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" (UID: "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:05:19.127855 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.127831    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-api-access-754k7" (OuterVolumeSpecName: "kube-api-access-754k7") pod "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" (UID: "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d"). InnerVolumeSpecName "kube-api-access-754k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:05:19.127855 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.127842    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" (UID: "f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:05:19.226568 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.226526    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:19.226568 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.226560    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:19.226568 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.226571    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-754k7\" (UniqueName: \"kubernetes.io/projected/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kube-api-access-754k7\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:19.226568 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.226580    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:19.226875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.226588    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:05:19.953355 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.953327    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"
Jun 15 06:05:19.953355 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.953336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8" event={"ID":"f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d","Type":"ContainerDied","Data":"2a720493f0227ae241ed17e815a327d6d77ce6859f6218c00c99ce19b0147a13"}
Jun 15 06:05:19.953927 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.953398    2579 scope.go:117] "RemoveContainer" containerID="cfd8b9a2ed5746092d8c22b6389b00bc6bedb2c43862705b1914253b1ad19b81"
Jun 15 06:05:19.962245 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.962224    2579 scope.go:117] "RemoveContainer" containerID="13a059c580f2e800c282161fc314167ce7e7fb399e0916408ab25c8a0ed8e6e2"
Jun 15 06:05:19.969501 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.969484    2579 scope.go:117] "RemoveContainer" containerID="ff2064b7f395788a0bafbbadc36250b64401bfe3d34d05d8fb2c8fa712bd8345"
Jun 15 06:05:19.975907 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.975882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"]
Jun 15 06:05:19.981390 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:19.981365    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-ea1cf-predictor-746cbfc697-2wbg8"]
Jun 15 06:05:20.159825 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:20.159789    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" path="/var/lib/kubelet/pods/f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d/volumes"
Jun 15 06:05:24.957904 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:24.957873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:05:33.600672 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.600578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:33.644265 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644228    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"]
Jun 15 06:05:33.644545 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644530    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="storage-initializer"
Jun 15 06:05:33.644545 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644546    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="storage-initializer"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644556    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644562    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644569    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kube-rbac-proxy"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644577    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kube-rbac-proxy"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644591    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="storage-initializer"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644596    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="storage-initializer"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644605    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644611    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644617    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kube-rbac-proxy"
Jun 15 06:05:33.644631 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644623    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kube-rbac-proxy"
Jun 15 06:05:33.644953 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644662    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kube-rbac-proxy"
Jun 15 06:05:33.644953 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644669    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kserve-container"
Jun 15 06:05:33.644953 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644676    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfeb06b4-e61d-4d7b-89a3-5095f6ac6ebf" containerName="kube-rbac-proxy"
Jun 15 06:05:33.644953 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.644681    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c43bf0-8ba9-49d7-8dc9-af68f3a2c07d" containerName="kserve-container"
Jun 15 06:05:33.647675 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.647656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.649874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.649813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-e564c-predictor-serving-cert\""
Jun 15 06:05:33.649874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.649862    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:05:33.650059 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.649819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jun 15 06:05:33.650059 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.649952    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-e564c-kube-rbac-proxy-sar-config\""
Jun 15 06:05:33.657588 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.657562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"]
Jun 15 06:05:33.721111 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.721080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:33.729218 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.729188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrjkm\" (UniqueName: \"kubernetes.io/projected/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-api-access-hrjkm\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.729330 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.729223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-cabundle-cert\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.729330 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.729245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.729397 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.729328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-proxy-tls\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.729397 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.729374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kserve-provision-location\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.830436 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.830392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-proxy-tls\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.830617 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.830447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kserve-provision-location\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.830617 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.830514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrjkm\" (UniqueName: \"kubernetes.io/projected/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-api-access-hrjkm\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.830617 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.830538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-cabundle-cert\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.830617 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.830596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.831000 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.830977    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kserve-provision-location\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.831281 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.831260    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-cabundle-cert\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.831359 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.831329    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.833001 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.832985    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-proxy-tls\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.840052 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.840030    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrjkm\" (UniqueName: \"kubernetes.io/projected/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-api-access-hrjkm\") pod \"isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:33.958815 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:33.958771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:34.103337 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:34.103310    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"]
Jun 15 06:05:34.105561 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:05:34.105534    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2bb2e7_2f13_4183_8c96_e8157ab031a3.slice/crio-f362ec8f6801d87ac15cccc9aae013533da8bb5e5fe8733e1ebce914e420ed07 WatchSource:0}: Error finding container f362ec8f6801d87ac15cccc9aae013533da8bb5e5fe8733e1ebce914e420ed07: Status 404 returned error can't find the container with id f362ec8f6801d87ac15cccc9aae013533da8bb5e5fe8733e1ebce914e420ed07
Jun 15 06:05:35.011830 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:35.011795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerStarted","Data":"5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0"}
Jun 15 06:05:35.012298 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:35.011859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerStarted","Data":"f362ec8f6801d87ac15cccc9aae013533da8bb5e5fe8733e1ebce914e420ed07"}
Jun 15 06:05:38.024100 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:38.024066    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerID="5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0" exitCode=0
Jun 15 06:05:38.024476 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:38.024142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerDied","Data":"5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0"}
Jun 15 06:05:39.034521 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:39.034477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerStarted","Data":"8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4"}
Jun 15 06:05:39.034928 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:39.034559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerStarted","Data":"cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48"}
Jun 15 06:05:39.034928 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:39.034586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerStarted","Data":"857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf"}
Jun 15 06:05:39.034928 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:39.034892    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:39.035066 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:39.035047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:39.036390 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:39.036360    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:05:39.054628 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:39.054579    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podStartSLOduration=2.343108387 podStartE2EDuration="6.054562263s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711453876s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:05:39.052997098 +0000 UTC m=+1077.373914406" watchObservedRunningTime="2026-06-15 06:05:39.054562263 +0000 UTC m=+1077.375479554"
Jun 15 06:05:40.038688 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:40.038662    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:40.039077 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:40.038802    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:05:40.039947 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:40.039919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:05:40.729712 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:40.729681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:05:41.042648 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:41.042553    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:05:41.043105 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:41.042966    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:05:46.048294 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:46.048265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:05:46.048912 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:46.048884    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:05:46.049229 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:46.049207    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:05:56.049437 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:56.049388    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:05:56.049881 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:05:56.049767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:06:06.049603 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:06.049554    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:06:06.050113 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:06.050042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:06:16.048892 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:16.048844    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:06:16.049402 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:16.049310    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:06:26.049645 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:26.049584    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:06:26.050160 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:26.050138    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:06:36.049803 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:36.049763    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:06:36.050254 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:36.050210    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:06:46.049804 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:46.049754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:06:46.050309 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:46.050162    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:06:56.049936 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:56.049901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:06:56.050418 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:06:56.050400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:07:08.660307 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.660228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l_9e18deea-398e-4c1a-9f4b-900af722df84/kserve-container/0.log"
Jun 15 06:07:08.867772 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.867713    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"]
Jun 15 06:07:08.868084 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.868033    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" containerID="cri-o://857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf" gracePeriod=30
Jun 15 06:07:08.868317 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.868062    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" containerID="cri-o://8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4" gracePeriod=30
Jun 15 06:07:08.868411 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.868091    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy" containerID="cri-o://cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48" gracePeriod=30
Jun 15 06:07:08.881766 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.881717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:07:08.936231 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.936150    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"]
Jun 15 06:07:08.939475 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.939452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:08.939942 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.939904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:07:08.941519 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.941496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5802e-kube-rbac-proxy-sar-config\""
Jun 15 06:07:08.941668 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.941648    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-5802e-predictor-serving-cert\""
Jun 15 06:07:08.953239 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.953212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"]
Jun 15 06:07:08.970884 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.970851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"]
Jun 15 06:07:08.971146 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.971120    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kserve-container" containerID="cri-o://1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8" gracePeriod=30
Jun 15 06:07:08.971276 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:08.971237    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kube-rbac-proxy" containerID="cri-o://81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248" gracePeriod=30
Jun 15 06:07:09.001092 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.001040    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:07:09.074282 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.074251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-proxy-tls\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.074440 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.074299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.074440 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.074324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.074440 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.074348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.074440 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.074387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pf6\" (UniqueName: \"kubernetes.io/projected/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-api-access-85pf6\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.175420 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.175384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pf6\" (UniqueName: \"kubernetes.io/projected/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-api-access-85pf6\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.175579 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.175432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-proxy-tls\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.175579 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.175478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.175579 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.175498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.175579 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.175532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.176016 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.175986    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.176341 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.176319    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.176499 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.176479    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-cabundle-cert\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.178400 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.178375    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-proxy-tls\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.183926 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.183849    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pf6\" (UniqueName: \"kubernetes.io/projected/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-api-access-85pf6\") pod \"isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.214288 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.214262    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:07:09.253978 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.253946    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:09.371971 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.371934    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerID="cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48" exitCode=2
Jun 15 06:07:09.372140 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.372007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerDied","Data":"cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48"}
Jun 15 06:07:09.373880 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.373853    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e18deea-398e-4c1a-9f4b-900af722df84" containerID="81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248" exitCode=2
Jun 15 06:07:09.373880 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.373876    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e18deea-398e-4c1a-9f4b-900af722df84" containerID="1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8" exitCode=2
Jun 15 06:07:09.374037 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.373905    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"
Jun 15 06:07:09.374037 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.373935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" event={"ID":"9e18deea-398e-4c1a-9f4b-900af722df84","Type":"ContainerDied","Data":"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"}
Jun 15 06:07:09.374037 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.373990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" event={"ID":"9e18deea-398e-4c1a-9f4b-900af722df84","Type":"ContainerDied","Data":"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"}
Jun 15 06:07:09.374037 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.374013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l" event={"ID":"9e18deea-398e-4c1a-9f4b-900af722df84","Type":"ContainerDied","Data":"6590dad55e38777339e94f188371442dd27b2e2f7b2841186790825480f9df18"}
Jun 15 06:07:09.374037 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.374039    2579 scope.go:117] "RemoveContainer" containerID="81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"
Jun 15 06:07:09.376866 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.376845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e18deea-398e-4c1a-9f4b-900af722df84-kube-rbac-proxy-sar-config\") pod \"9e18deea-398e-4c1a-9f4b-900af722df84\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") "
Jun 15 06:07:09.376968 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.376892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x582m\" (UniqueName: \"kubernetes.io/projected/9e18deea-398e-4c1a-9f4b-900af722df84-kube-api-access-x582m\") pod \"9e18deea-398e-4c1a-9f4b-900af722df84\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") "
Jun 15 06:07:09.376968 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.376946    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls\") pod \"9e18deea-398e-4c1a-9f4b-900af722df84\" (UID: \"9e18deea-398e-4c1a-9f4b-900af722df84\") "
Jun 15 06:07:09.377291 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.377262    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e18deea-398e-4c1a-9f4b-900af722df84-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e18deea-398e-4c1a-9f4b-900af722df84" (UID: "9e18deea-398e-4c1a-9f4b-900af722df84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:07:09.379103 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.379081    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e18deea-398e-4c1a-9f4b-900af722df84" (UID: "9e18deea-398e-4c1a-9f4b-900af722df84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:07:09.379274 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.379255    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e18deea-398e-4c1a-9f4b-900af722df84-kube-api-access-x582m" (OuterVolumeSpecName: "kube-api-access-x582m") pod "9e18deea-398e-4c1a-9f4b-900af722df84" (UID: "9e18deea-398e-4c1a-9f4b-900af722df84"). InnerVolumeSpecName "kube-api-access-x582m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:07:09.382100 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.382082    2579 scope.go:117] "RemoveContainer" containerID="1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"
Jun 15 06:07:09.387636 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.387616    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"]
Jun 15 06:07:09.390080 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.390059    2579 scope.go:117] "RemoveContainer" containerID="81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"
Jun 15 06:07:09.390328 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:07:09.390310    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248\": container with ID starting with 81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248 not found: ID does not exist" containerID="81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"
Jun 15 06:07:09.390403 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.390341    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"} err="failed to get container status \"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248\": rpc error: code = NotFound desc = could not find container \"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248\": container with ID starting with 81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248 not found: ID does not exist"
Jun 15 06:07:09.390403 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.390366    2579 scope.go:117] "RemoveContainer" containerID="1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"
Jun 15 06:07:09.390609 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:07:09.390591    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8\": container with ID starting with 1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8 not found: ID does not exist" containerID="1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"
Jun 15 06:07:09.390683 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.390619    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"} err="failed to get container status \"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8\": rpc error: code = NotFound desc = could not find container \"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8\": container with ID starting with 1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8 not found: ID does not exist"
Jun 15 06:07:09.390683 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.390637    2579 scope.go:117] "RemoveContainer" containerID="81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"
Jun 15 06:07:09.390941 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.390914    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248"} err="failed to get container status \"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248\": rpc error: code = NotFound desc = could not find container \"81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248\": container with ID starting with 81c68ee8426cd85623bff1a4591175fd8783bb1ae8997b1c1be4b92a051f3248 not found: ID does not exist"
Jun 15 06:07:09.390988 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.390943    2579 scope.go:117] "RemoveContainer" containerID="1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"
Jun 15 06:07:09.391177 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.391147    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8"} err="failed to get container status \"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8\": rpc error: code = NotFound desc = could not find container \"1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8\": container with ID starting with 1b907df19172fd1cfd16a09a6bcf90a8d87b2aaaf8851a8cb26beb563766c8f8 not found: ID does not exist"
Jun 15 06:07:09.391500 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:07:09.391478    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f10b72_a7b4_40aa_bed4_8e3b2c06ee12.slice/crio-0a6c330cc93b70a7a075ae5a2b94031740cfce6c168893793959698c7efe2d93 WatchSource:0}: Error finding container 0a6c330cc93b70a7a075ae5a2b94031740cfce6c168893793959698c7efe2d93: Status 404 returned error can't find the container with id 0a6c330cc93b70a7a075ae5a2b94031740cfce6c168893793959698c7efe2d93
Jun 15 06:07:09.478125 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.478047    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e18deea-398e-4c1a-9f4b-900af722df84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:09.478125 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.478073    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x582m\" (UniqueName: \"kubernetes.io/projected/9e18deea-398e-4c1a-9f4b-900af722df84-kube-api-access-x582m\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:09.478125 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.478085    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e18deea-398e-4c1a-9f4b-900af722df84-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:09.696386 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.696350    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"]
Jun 15 06:07:09.700328 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:09.700299    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-e564c-predictor-5478dc4bc7-9nf6l"]
Jun 15 06:07:10.158758 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:10.158669    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" path="/var/lib/kubelet/pods/9e18deea-398e-4c1a-9f4b-900af722df84/volumes"
Jun 15 06:07:10.380473 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:10.380441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerStarted","Data":"6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788"}
Jun 15 06:07:10.380652 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:10.380494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerStarted","Data":"0a6c330cc93b70a7a075ae5a2b94031740cfce6c168893793959698c7efe2d93"}
Jun 15 06:07:11.043483 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:11.043438    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jun 15 06:07:13.394434 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:13.394398    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerID="6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788" exitCode=0
Jun 15 06:07:13.394835 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:13.394459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerDied","Data":"6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788"}
Jun 15 06:07:14.400779 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:14.400745    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerID="857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf" exitCode=0
Jun 15 06:07:14.401212 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:14.400754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerDied","Data":"857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf"}
Jun 15 06:07:14.402885 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:14.402859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerStarted","Data":"62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4"}
Jun 15 06:07:14.402992 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:14.402914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerStarted","Data":"f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f"}
Jun 15 06:07:14.403119 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:14.403100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:14.421774 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:14.421700    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podStartSLOduration=2.527376231 podStartE2EDuration="6.421683279s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.894307048s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:07:14.420019307 +0000 UTC m=+1172.740936593" watchObservedRunningTime="2026-06-15 06:07:14.421683279 +0000 UTC m=+1172.742600621"
Jun 15 06:07:15.406492 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:15.406459    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:15.407695 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:15.407667    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:07:16.043661 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:16.043613    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jun 15 06:07:16.049509 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:16.049419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:07:16.051250 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:16.051221    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:07:16.410486 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:16.410439    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:07:21.043070 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:21.043021    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jun 15 06:07:21.043512 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:21.043145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:07:21.414790 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:21.414760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:07:21.415277 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:21.415246    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:07:26.043195 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:26.043146    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jun 15 06:07:26.049220 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:26.049186    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:07:26.050819 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:26.050792    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:07:31.042949 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:31.042902    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jun 15 06:07:31.415328 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:31.415283    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:07:36.043033 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:36.042986    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jun 15 06:07:36.049461 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:36.049433    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jun 15 06:07:36.049558 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:36.049546    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:07:36.051083 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:36.051050    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:07:36.051199 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:36.051183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:07:39.017280 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.017255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:07:39.103433 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.103378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-cabundle-cert\") pod \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") "
Jun 15 06:07:39.103615 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.103463    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrjkm\" (UniqueName: \"kubernetes.io/projected/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-api-access-hrjkm\") pod \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") "
Jun 15 06:07:39.103615 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.103546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-rbac-proxy-sar-config\") pod \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") "
Jun 15 06:07:39.103771 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.103651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-proxy-tls\") pod \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") "
Jun 15 06:07:39.103771 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.103707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kserve-provision-location\") pod \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\" (UID: \"dd2bb2e7-2f13-4183-8c96-e8157ab031a3\") "
Jun 15 06:07:39.103861 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.103820    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd2bb2e7-2f13-4183-8c96-e8157ab031a3" (UID: "dd2bb2e7-2f13-4183-8c96-e8157ab031a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:07:39.103899 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.103861    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd2bb2e7-2f13-4183-8c96-e8157ab031a3" (UID: "dd2bb2e7-2f13-4183-8c96-e8157ab031a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:07:39.104063 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.104037    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd2bb2e7-2f13-4183-8c96-e8157ab031a3" (UID: "dd2bb2e7-2f13-4183-8c96-e8157ab031a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:07:39.104132 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.104047    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:39.104132 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.104085    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:39.105632 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.105613    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-api-access-hrjkm" (OuterVolumeSpecName: "kube-api-access-hrjkm") pod "dd2bb2e7-2f13-4183-8c96-e8157ab031a3" (UID: "dd2bb2e7-2f13-4183-8c96-e8157ab031a3"). InnerVolumeSpecName "kube-api-access-hrjkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:07:39.105850 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.105826    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd2bb2e7-2f13-4183-8c96-e8157ab031a3" (UID: "dd2bb2e7-2f13-4183-8c96-e8157ab031a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:07:39.205170 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.205072    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrjkm\" (UniqueName: \"kubernetes.io/projected/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kube-api-access-hrjkm\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:39.205170 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.205101    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:39.205170 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.205115    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd2bb2e7-2f13-4183-8c96-e8157ab031a3-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:07:39.499055 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.498965    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerID="8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4" exitCode=0
Jun 15 06:07:39.499055 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.499026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"
Jun 15 06:07:39.499243 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.499051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerDied","Data":"8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4"}
Jun 15 06:07:39.499243 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.499125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2" event={"ID":"dd2bb2e7-2f13-4183-8c96-e8157ab031a3","Type":"ContainerDied","Data":"f362ec8f6801d87ac15cccc9aae013533da8bb5e5fe8733e1ebce914e420ed07"}
Jun 15 06:07:39.499243 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.499157    2579 scope.go:117] "RemoveContainer" containerID="8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4"
Jun 15 06:07:39.507805 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.507784    2579 scope.go:117] "RemoveContainer" containerID="cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48"
Jun 15 06:07:39.515238 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.515221    2579 scope.go:117] "RemoveContainer" containerID="857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf"
Jun 15 06:07:39.521597 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.521572    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"]
Jun 15 06:07:39.523424 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.523409    2579 scope.go:117] "RemoveContainer" containerID="5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0"
Jun 15 06:07:39.527263 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.527241    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-e564c-predictor-566bdb64cc-mnsp2"]
Jun 15 06:07:39.530834 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.530813    2579 scope.go:117] "RemoveContainer" containerID="8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4"
Jun 15 06:07:39.531099 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:07:39.531081    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4\": container with ID starting with 8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4 not found: ID does not exist" containerID="8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4"
Jun 15 06:07:39.531151 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.531109    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4"} err="failed to get container status \"8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4\": rpc error: code = NotFound desc = could not find container \"8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4\": container with ID starting with 8a836125308a2271d122986f7195c298cc3909ab9d2dfb0722f7dad52b77c8e4 not found: ID does not exist"
Jun 15 06:07:39.531151 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.531128    2579 scope.go:117] "RemoveContainer" containerID="cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48"
Jun 15 06:07:39.531409 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:07:39.531391    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48\": container with ID starting with cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48 not found: ID does not exist" containerID="cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48"
Jun 15 06:07:39.531461 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.531417    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48"} err="failed to get container status \"cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48\": rpc error: code = NotFound desc = could not find container \"cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48\": container with ID starting with cb79792e607e9fd278110606f955aaa2f6c28d20fe9ca5151c8d40577e4dcf48 not found: ID does not exist"
Jun 15 06:07:39.531461 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.531433    2579 scope.go:117] "RemoveContainer" containerID="857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf"
Jun 15 06:07:39.531642 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:07:39.531627    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf\": container with ID starting with 857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf not found: ID does not exist" containerID="857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf"
Jun 15 06:07:39.531685 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.531648    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf"} err="failed to get container status \"857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf\": rpc error: code = NotFound desc = could not find container \"857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf\": container with ID starting with 857078df243d5b63a83f194d032193ed9cfdff5f431770bd09a2db6fec00dfaf not found: ID does not exist"
Jun 15 06:07:39.531685 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.531662    2579 scope.go:117] "RemoveContainer" containerID="5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0"
Jun 15 06:07:39.531917 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:07:39.531900    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0\": container with ID starting with 5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0 not found: ID does not exist" containerID="5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0"
Jun 15 06:07:39.531959 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:39.531921    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0"} err="failed to get container status \"5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0\": rpc error: code = NotFound desc = could not find container \"5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0\": container with ID starting with 5cb3f8c000604bcb10755c6fba8f6ab174c84f93e93844e14d52d52e6a0819c0 not found: ID does not exist"
Jun 15 06:07:40.164303 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:40.164258    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" path="/var/lib/kubelet/pods/dd2bb2e7-2f13-4183-8c96-e8157ab031a3/volumes"
Jun 15 06:07:41.415312 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:41.415268    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:07:42.177952 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:42.177928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 06:07:42.180874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:42.180848    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 06:07:51.415321 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:07:51.415281    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:08:01.415707 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:08:01.415657    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:08:09.146444 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:08:09.146406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:08:11.416169 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:08:11.416133    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:08:21.415419 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:08:21.415374    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:08:30.154538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:08:30.154433    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:08:40.154636 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:08:40.154594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:08:50.154938 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:08:50.154887    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:09:00.155405 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:00.155350    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:09:10.154719 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:10.154669    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:09:20.155132 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:20.155078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:09:30.155176 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:30.155116    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:09:40.154454 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:40.154409    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:09:50.158395 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:50.158365    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:09:59.110601 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.110522    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"]
Jun 15 06:09:59.111084 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.110822    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" containerID="cri-o://f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f" gracePeriod=30
Jun 15 06:09:59.111084 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.110896    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kube-rbac-proxy" containerID="cri-o://62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4" gracePeriod=30
Jun 15 06:09:59.162648 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.162616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:09:59.206452 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206403    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"]
Jun 15 06:09:59.206781 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206762    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206782    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206794    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kube-rbac-proxy"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206802    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kube-rbac-proxy"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206816    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kserve-container"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206825    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kserve-container"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206842    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="storage-initializer"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206850    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="storage-initializer"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206867    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy"
Jun 15 06:09:59.206875 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy"
Jun 15 06:09:59.207320 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206888    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container"
Jun 15 06:09:59.207320 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206896    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container"
Jun 15 06:09:59.207320 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206967    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kserve-container"
Jun 15 06:09:59.207320 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206980    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kube-rbac-proxy"
Jun 15 06:09:59.207320 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.206990    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="agent"
Jun 15 06:09:59.207320 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.207000    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e18deea-398e-4c1a-9f4b-900af722df84" containerName="kserve-container"
Jun 15 06:09:59.207320 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.207009    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2bb2e7-2f13-4183-8c96-e8157ab031a3" containerName="kube-rbac-proxy"
Jun 15 06:09:59.210310 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.210288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.212610 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.212583    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f19eac-kube-rbac-proxy-sar-config\""
Jun 15 06:09:59.212718 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.212597    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f19eac-predictor-serving-cert\""
Jun 15 06:09:59.218411 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.218385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"]
Jun 15 06:09:59.256551 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.256520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-cabundle-cert\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.256769 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.256560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.256769 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.256610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.256769 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.256671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5nh\" (UniqueName: \"kubernetes.io/projected/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-api-access-qz5nh\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.256769 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.256752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kserve-provision-location\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.357577 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.357532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.357821 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.357587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.357821 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.357618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5nh\" (UniqueName: \"kubernetes.io/projected/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-api-access-qz5nh\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.357821 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.357655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kserve-provision-location\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.357821 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:09:59.357695    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-f19eac-predictor-serving-cert: secret "isvc-primary-f19eac-predictor-serving-cert" not found
Jun 15 06:09:59.357821 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:09:59.357795    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls podName:98f8fee9-da67-4069-a0e4-738cb7dc7a9c nodeName:}" failed. No retries permitted until 2026-06-15 06:09:59.857774838 +0000 UTC m=+1338.178692118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls") pod "isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" (UID: "98f8fee9-da67-4069-a0e4-738cb7dc7a9c") : secret "isvc-primary-f19eac-predictor-serving-cert" not found
Jun 15 06:09:59.357821 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.357817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-cabundle-cert\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.358153 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.358076    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kserve-provision-location\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.358415 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.358398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-cabundle-cert\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.358415 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.358407    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.366793 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.366704    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5nh\" (UniqueName: \"kubernetes.io/projected/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-api-access-qz5nh\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.860381 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.860350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:09:59.862833 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:09:59.862800    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls\") pod \"isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") " pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:10:00.007397 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:00.007356    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerID="62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4" exitCode=2
Jun 15 06:10:00.007397 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:00.007395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerDied","Data":"62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4"}
Jun 15 06:10:00.121782 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:00.121666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:10:00.127974 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:00.127950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:10:00.155012 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:00.154969    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jun 15 06:10:00.253024 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:00.252842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"]
Jun 15 06:10:00.255642 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:10:00.255601    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f8fee9_da67_4069_a0e4_738cb7dc7a9c.slice/crio-9a1908eb6a19b4c0c058f59fd0c0d13b10b21698e2598720429ab8849b5f2c6f WatchSource:0}: Error finding container 9a1908eb6a19b4c0c058f59fd0c0d13b10b21698e2598720429ab8849b5f2c6f: Status 404 returned error can't find the container with id 9a1908eb6a19b4c0c058f59fd0c0d13b10b21698e2598720429ab8849b5f2c6f
Jun 15 06:10:01.012845 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:01.012803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerStarted","Data":"7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7"}
Jun 15 06:10:01.013033 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:01.012866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerStarted","Data":"9a1908eb6a19b4c0c058f59fd0c0d13b10b21698e2598720429ab8849b5f2c6f"}
Jun 15 06:10:01.411120 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:01.411077    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jun 15 06:10:05.031233 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:05.031197    2579 generic.go:356] "Generic (PLEG): container finished" podID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerID="7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7" exitCode=0
Jun 15 06:10:05.031614 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:05.031273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerDied","Data":"7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7"}
Jun 15 06:10:06.037703 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:06.037661    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerStarted","Data":"68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83"}
Jun 15 06:10:06.038208 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:06.037749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerStarted","Data":"26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d"}
Jun 15 06:10:06.038208 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:06.037989    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:10:06.056788 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:06.056710    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podStartSLOduration=3.179036703 podStartE2EDuration="7.056692162s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877655459s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:10:06.055471746 +0000 UTC m=+1344.376389031" watchObservedRunningTime="2026-06-15 06:10:06.056692162 +0000 UTC m=+1344.377609451"
Jun 15 06:10:06.411592 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:06.411547    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jun 15 06:10:07.041690 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:07.041661    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:10:07.042950 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:07.042920    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:10:08.045361 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:08.045315    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:10:09.773639 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.773573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:10:09.836625 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.836591    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-proxy-tls\") pod \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") "
Jun 15 06:10:09.836831 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.836634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-rbac-proxy-sar-config\") pod \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") "
Jun 15 06:10:09.836831 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.836664    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-cabundle-cert\") pod \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") "
Jun 15 06:10:09.836831 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.836719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kserve-provision-location\") pod \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") "
Jun 15 06:10:09.836831 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.836764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85pf6\" (UniqueName: \"kubernetes.io/projected/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-api-access-85pf6\") pod \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\" (UID: \"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12\") "
Jun 15 06:10:09.837096 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.837068    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" (UID: "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:10:09.837144 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.837113    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" (UID: "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:10:09.837224 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.837201    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" (UID: "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:10:09.838859 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.838831    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-api-access-85pf6" (OuterVolumeSpecName: "kube-api-access-85pf6") pod "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" (UID: "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12"). InnerVolumeSpecName "kube-api-access-85pf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:10:09.838859 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.838847    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" (UID: "f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:10:09.938197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.938158    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:10:09.938197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.938190    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:10:09.938197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.938204    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85pf6\" (UniqueName: \"kubernetes.io/projected/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-api-access-85pf6\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:10:09.938447 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.938217    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:10:09.938447 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:09.938229    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:10:10.053973 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.053882    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerID="f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f" exitCode=0
Jun 15 06:10:10.053973 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.053945    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"
Jun 15 06:10:10.053973 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.053961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerDied","Data":"f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f"}
Jun 15 06:10:10.054213 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.054033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb" event={"ID":"f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12","Type":"ContainerDied","Data":"0a6c330cc93b70a7a075ae5a2b94031740cfce6c168893793959698c7efe2d93"}
Jun 15 06:10:10.054213 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.054073    2579 scope.go:117] "RemoveContainer" containerID="62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4"
Jun 15 06:10:10.062940 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.062917    2579 scope.go:117] "RemoveContainer" containerID="f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f"
Jun 15 06:10:10.070483 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.070451    2579 scope.go:117] "RemoveContainer" containerID="6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788"
Jun 15 06:10:10.077446 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.077418    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"]
Jun 15 06:10:10.080352 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.080330    2579 scope.go:117] "RemoveContainer" containerID="62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4"
Jun 15 06:10:10.081184 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:10:10.081148    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4\": container with ID starting with 62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4 not found: ID does not exist" containerID="62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4"
Jun 15 06:10:10.081307 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.081196    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4"} err="failed to get container status \"62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4\": rpc error: code = NotFound desc = could not find container \"62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4\": container with ID starting with 62d3e7c17bf132f8899dfb0f1c3f453754f87d1a3c821f61f3812fe458b885e4 not found: ID does not exist"
Jun 15 06:10:10.081307 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.081222    2579 scope.go:117] "RemoveContainer" containerID="f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f"
Jun 15 06:10:10.081535 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:10:10.081516    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f\": container with ID starting with f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f not found: ID does not exist" containerID="f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f"
Jun 15 06:10:10.081594 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.081542    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f"} err="failed to get container status \"f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f\": rpc error: code = NotFound desc = could not find container \"f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f\": container with ID starting with f606eb9e8622726a89638e4bc58e9668952b31893dad85bd7201ade932d1e52f not found: ID does not exist"
Jun 15 06:10:10.081594 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.081559    2579 scope.go:117] "RemoveContainer" containerID="6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788"
Jun 15 06:10:10.081841 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:10:10.081824    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788\": container with ID starting with 6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788 not found: ID does not exist" containerID="6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788"
Jun 15 06:10:10.081907 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.081845    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788"} err="failed to get container status \"6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788\": rpc error: code = NotFound desc = could not find container \"6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788\": container with ID starting with 6c2db542bff4f2384cefed2cfa67c5fd410c3ed13708b81004fd2d42fb02a788 not found: ID does not exist"
Jun 15 06:10:10.082351 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.082332    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-5802e-predictor-5589cdf8c-fqmlb"]
Jun 15 06:10:10.159741 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:10.159698    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" path="/var/lib/kubelet/pods/f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12/volumes"
Jun 15 06:10:13.049904 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:13.049873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:10:13.050391 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:13.050364    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:10:23.050824 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:23.050776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:10:33.051237 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:33.051192    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:10:43.050763 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:43.050708    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:10:53.051235 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:10:53.051186    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:11:03.050458 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:03.050419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:11:13.050489 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:13.050450    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jun 15 06:11:23.050878 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:23.050846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:11:29.363957 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.363880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:11:29.414787 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.414747    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"]
Jun 15 06:11:29.415038 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415024    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container"
Jun 15 06:11:29.415038 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container"
Jun 15 06:11:29.415128 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415048    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kube-rbac-proxy"
Jun 15 06:11:29.415128 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415054    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kube-rbac-proxy"
Jun 15 06:11:29.415128 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415064    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="storage-initializer"
Jun 15 06:11:29.415128 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415070    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="storage-initializer"
Jun 15 06:11:29.415128 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415126    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kserve-container"
Jun 15 06:11:29.415294 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.415135    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f10b72-a7b4-40aa-bed4-8e3b2c06ee12" containerName="kube-rbac-proxy"
Jun 15 06:11:29.417913 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.417889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.420154 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.420097    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f19eac-kube-rbac-proxy-sar-config\""
Jun 15 06:11:29.420274 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.420102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-f19eac-dockercfg-mvntd\""
Jun 15 06:11:29.420330 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.420299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f19eac-predictor-serving-cert\""
Jun 15 06:11:29.420387 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.420306    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-f19eac\""
Jun 15 06:11:29.421431 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.421094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ghl\" (UniqueName: \"kubernetes.io/projected/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-api-access-j5ghl\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.421431 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.421138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.421431 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.421268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-cabundle-cert\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.421431 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.421309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kserve-provision-location\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.421431 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.421375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.426953 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.426932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"]
Jun 15 06:11:29.522393 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.522357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.522615 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.522416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ghl\" (UniqueName: \"kubernetes.io/projected/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-api-access-j5ghl\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.522615 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.522446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.522615 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.522471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-cabundle-cert\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.522615 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.522500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kserve-provision-location\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.522886 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:29.522617    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-serving-cert: secret "isvc-secondary-f19eac-predictor-serving-cert" not found
Jun 15 06:11:29.522886 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:29.522747    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls podName:fb9e63f1-cdeb-4e5b-9952-b48937fb1e74 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:30.022704125 +0000 UTC m=+1428.343621391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls") pod "isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" (UID: "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74") : secret "isvc-secondary-f19eac-predictor-serving-cert" not found
Jun 15 06:11:29.523058 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.523036    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kserve-provision-location\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.523232 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.523213    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-cabundle-cert\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.523298 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.523256    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:29.533715 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:29.533688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ghl\" (UniqueName: \"kubernetes.io/projected/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-api-access-j5ghl\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:30.026803 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:30.026755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:30.026996 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:30.026929    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-serving-cert: secret "isvc-secondary-f19eac-predictor-serving-cert" not found
Jun 15 06:11:30.027040 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:30.027007    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls podName:fb9e63f1-cdeb-4e5b-9952-b48937fb1e74 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:31.026992126 +0000 UTC m=+1429.347909390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls") pod "isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" (UID: "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74") : secret "isvc-secondary-f19eac-predictor-serving-cert" not found
Jun 15 06:11:30.681860 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:30.681819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:11:31.034760 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:31.034654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:31.037183 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:31.037158    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls\") pod \"isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") " pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:31.229282 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:31.229248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:31.360564 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:31.360383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"]
Jun 15 06:11:31.363311 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:11:31.363287    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9e63f1_cdeb_4e5b_9952_b48937fb1e74.slice/crio-239b680b3a8806960926a5269ad02629390d0ddde8671c57c3a75d715970a386 WatchSource:0}: Error finding container 239b680b3a8806960926a5269ad02629390d0ddde8671c57c3a75d715970a386: Status 404 returned error can't find the container with id 239b680b3a8806960926a5269ad02629390d0ddde8671c57c3a75d715970a386
Jun 15 06:11:32.352394 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:32.352358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" event={"ID":"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74","Type":"ContainerStarted","Data":"51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2"}
Jun 15 06:11:32.352813 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:32.352426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" event={"ID":"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74","Type":"ContainerStarted","Data":"239b680b3a8806960926a5269ad02629390d0ddde8671c57c3a75d715970a386"}
Jun 15 06:11:35.364125 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:35.364096    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/storage-initializer/0.log"
Jun 15 06:11:35.364563 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:35.364134    2579 generic.go:356] "Generic (PLEG): container finished" podID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerID="51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2" exitCode=1
Jun 15 06:11:35.364563 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:35.364211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" event={"ID":"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74","Type":"ContainerDied","Data":"51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2"}
Jun 15 06:11:36.369902 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:36.369872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/storage-initializer/0.log"
Jun 15 06:11:36.370327 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:36.369949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" event={"ID":"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74","Type":"ContainerStarted","Data":"a30d0bcb06cd8f2665aa9730523332ed23d08424074bcfa1dd83c79732c3b1d4"}
Jun 15 06:11:42.395254 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:42.395224    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/storage-initializer/1.log"
Jun 15 06:11:42.395682 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:42.395623    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/storage-initializer/0.log"
Jun 15 06:11:42.395682 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:42.395656    2579 generic.go:356] "Generic (PLEG): container finished" podID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerID="a30d0bcb06cd8f2665aa9730523332ed23d08424074bcfa1dd83c79732c3b1d4" exitCode=1
Jun 15 06:11:42.395820 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:42.395705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" event={"ID":"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74","Type":"ContainerDied","Data":"a30d0bcb06cd8f2665aa9730523332ed23d08424074bcfa1dd83c79732c3b1d4"}
Jun 15 06:11:42.395820 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:42.395782    2579 scope.go:117] "RemoveContainer" containerID="51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2"
Jun 15 06:11:42.396174 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:42.396153    2579 scope.go:117] "RemoveContainer" containerID="51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2"
Jun 15 06:11:42.406339 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:42.406313    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_kserve-ci-e2e-test(fb9e63f1-cdeb-4e5b-9952-b48937fb1e74)\"" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74"
Jun 15 06:11:42.406511 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:42.406482    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_kserve-ci-e2e-test_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74_0 in pod sandbox 239b680b3a8806960926a5269ad02629390d0ddde8671c57c3a75d715970a386 from index: no such id: '51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2'" containerID="51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2"
Jun 15 06:11:42.406564 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:42.406521    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_kserve-ci-e2e-test_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74_0 in pod sandbox 239b680b3a8806960926a5269ad02629390d0ddde8671c57c3a75d715970a386 from index: no such id: '51ce7ff2dcb74fe0b9984c0a56e5aa90e173933fc52ae77fa0b75c1ccf0864c2'"
Jun 15 06:11:43.400476 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:43.400447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/storage-initializer/1.log"
Jun 15 06:11:47.464548 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.464511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"]
Jun 15 06:11:47.493932 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.493892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:11:47.520839 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.520807    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"]
Jun 15 06:11:47.521536 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.521339    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container" containerID="cri-o://26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d" gracePeriod=30
Jun 15 06:11:47.521536 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.521501    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kube-rbac-proxy" containerID="cri-o://68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83" gracePeriod=30
Jun 15 06:11:47.589991 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.589937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:11:47.621803 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.621763    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"]
Jun 15 06:11:47.626517 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.626492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.628747 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.628709    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4ed3d4-predictor-serving-cert\""
Jun 15 06:11:47.628872 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.628849    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4ed3d4-kube-rbac-proxy-sar-config\""
Jun 15 06:11:47.629039 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.629022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-4ed3d4-dockercfg-grgpv\""
Jun 15 06:11:47.629126 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.629067    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-4ed3d4\""
Jun 15 06:11:47.634904 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.634872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"]
Jun 15 06:11:47.650713 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.650689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.650874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.650744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a0ecfe-5f06-4878-a71f-8ac19e112e80-proxy-tls\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.650874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.650782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-cabundle-cert\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.650874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.650836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kserve-provision-location\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.651027 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.650888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgzm5\" (UniqueName: \"kubernetes.io/projected/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-api-access-zgzm5\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.655533 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.655510    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/storage-initializer/1.log"
Jun 15 06:11:47.657716 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.657698    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:47.695697 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.695666    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:11:47.751971 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.751881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5ghl\" (UniqueName: \"kubernetes.io/projected/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-api-access-j5ghl\") pod \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") "
Jun 15 06:11:47.751971 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.751925    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kserve-provision-location\") pod \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") "
Jun 15 06:11:47.751971 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.751974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls\") pod \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") "
Jun 15 06:11:47.752255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.751997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-rbac-proxy-sar-config\") pod \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") "
Jun 15 06:11:47.752255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-cabundle-cert\") pod \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\" (UID: \"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74\") "
Jun 15 06:11:47.752255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kserve-provision-location\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.752255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzm5\" (UniqueName: \"kubernetes.io/projected/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-api-access-zgzm5\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.752255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.752255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a0ecfe-5f06-4878-a71f-8ac19e112e80-proxy-tls\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.752255 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-cabundle-cert\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.752605 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752363    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" (UID: "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:47.752605 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752430    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" (UID: "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:47.752605 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752484    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" (UID: "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:11:47.752605 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.752573    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kserve-provision-location\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.753074 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.753045    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-cabundle-cert\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.753186 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.753169    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.754331 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.754304    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-api-access-j5ghl" (OuterVolumeSpecName: "kube-api-access-j5ghl") pod "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" (UID: "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74"). InnerVolumeSpecName "kube-api-access-j5ghl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:47.754487 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.754471    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" (UID: "fb9e63f1-cdeb-4e5b-9952-b48937fb1e74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:47.754667 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.754652    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a0ecfe-5f06-4878-a71f-8ac19e112e80-proxy-tls\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.761048 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.761028    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzm5\" (UniqueName: \"kubernetes.io/projected/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-api-access-zgzm5\") pod \"isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") " pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:47.852886 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.852838    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:47.852886 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.852882    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:47.852886 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.852893    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5ghl\" (UniqueName: \"kubernetes.io/projected/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kube-api-access-j5ghl\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:47.852886 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.852901    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:47.853171 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.852912    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:47.952868 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:47.952832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:48.046509 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.046463    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jun 15 06:11:48.087175 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.087110    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"]
Jun 15 06:11:48.089332 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:11:48.089307    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a0ecfe_5f06_4878_a71f_8ac19e112e80.slice/crio-75c812927e2a3098cb98763d01e65e0ae500a06fc3d21040998bcbd2eb8cf225 WatchSource:0}: Error finding container 75c812927e2a3098cb98763d01e65e0ae500a06fc3d21040998bcbd2eb8cf225: Status 404 returned error can't find the container with id 75c812927e2a3098cb98763d01e65e0ae500a06fc3d21040998bcbd2eb8cf225
Jun 15 06:11:48.421470 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.421434    2579 generic.go:356] "Generic (PLEG): container finished" podID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerID="68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83" exitCode=2
Jun 15 06:11:48.421675 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.421488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerDied","Data":"68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83"}
Jun 15 06:11:48.422912 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.422892    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml_fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/storage-initializer/1.log"
Jun 15 06:11:48.423007 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.422995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"
Jun 15 06:11:48.423007 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.422998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml" event={"ID":"fb9e63f1-cdeb-4e5b-9952-b48937fb1e74","Type":"ContainerDied","Data":"239b680b3a8806960926a5269ad02629390d0ddde8671c57c3a75d715970a386"}
Jun 15 06:11:48.423129 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.423064    2579 scope.go:117] "RemoveContainer" containerID="a30d0bcb06cd8f2665aa9730523332ed23d08424074bcfa1dd83c79732c3b1d4"
Jun 15 06:11:48.424742 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.424700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh" event={"ID":"11a0ecfe-5f06-4878-a71f-8ac19e112e80","Type":"ContainerStarted","Data":"da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece"}
Jun 15 06:11:48.424851 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.424772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh" event={"ID":"11a0ecfe-5f06-4878-a71f-8ac19e112e80","Type":"ContainerStarted","Data":"75c812927e2a3098cb98763d01e65e0ae500a06fc3d21040998bcbd2eb8cf225"}
Jun 15 06:11:48.457167 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.457132    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"]
Jun 15 06:11:48.461176 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:48.461150    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f19eac-predictor-6d97b7b459-hs9ml"]
Jun 15 06:11:50.159137 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:50.159106    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" path="/var/lib/kubelet/pods/fb9e63f1-cdeb-4e5b-9952-b48937fb1e74/volumes"
Jun 15 06:11:51.437581 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:51.437548    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh_11a0ecfe-5f06-4878-a71f-8ac19e112e80/storage-initializer/0.log"
Jun 15 06:11:51.438045 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:51.437589    2579 generic.go:356] "Generic (PLEG): container finished" podID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerID="da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece" exitCode=1
Jun 15 06:11:51.438045 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:51.437669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh" event={"ID":"11a0ecfe-5f06-4878-a71f-8ac19e112e80","Type":"ContainerDied","Data":"da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece"}
Jun 15 06:11:52.443211 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.443190    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh_11a0ecfe-5f06-4878-a71f-8ac19e112e80/storage-initializer/0.log"
Jun 15 06:11:52.443535 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.443273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh" event={"ID":"11a0ecfe-5f06-4878-a71f-8ac19e112e80","Type":"ContainerStarted","Data":"4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72"}
Jun 15 06:11:52.579247 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.579208    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"]
Jun 15 06:11:52.594056 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.594028    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:11:52.690529 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.690431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kserve-provision-location\") pod \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") "
Jun 15 06:11:52.690529 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.690476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls\") pod \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") "
Jun 15 06:11:52.690529 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.690510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-cabundle-cert\") pod \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") "
Jun 15 06:11:52.690529 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.690529    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz5nh\" (UniqueName: \"kubernetes.io/projected/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-api-access-qz5nh\") pod \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") "
Jun 15 06:11:52.690910 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.690553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-rbac-proxy-sar-config\") pod \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\" (UID: \"98f8fee9-da67-4069-a0e4-738cb7dc7a9c\") "
Jun 15 06:11:52.690910 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.690803    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98f8fee9-da67-4069-a0e4-738cb7dc7a9c" (UID: "98f8fee9-da67-4069-a0e4-738cb7dc7a9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:11:52.691016 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.690959    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98f8fee9-da67-4069-a0e4-738cb7dc7a9c" (UID: "98f8fee9-da67-4069-a0e4-738cb7dc7a9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:52.691133 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.691109    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98f8fee9-da67-4069-a0e4-738cb7dc7a9c" (UID: "98f8fee9-da67-4069-a0e4-738cb7dc7a9c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:52.692864 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.692831    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-api-access-qz5nh" (OuterVolumeSpecName: "kube-api-access-qz5nh") pod "98f8fee9-da67-4069-a0e4-738cb7dc7a9c" (UID: "98f8fee9-da67-4069-a0e4-738cb7dc7a9c"). InnerVolumeSpecName "kube-api-access-qz5nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:52.692968 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.692893    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98f8fee9-da67-4069-a0e4-738cb7dc7a9c" (UID: "98f8fee9-da67-4069-a0e4-738cb7dc7a9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:52.711204 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.711168    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"]
Jun 15 06:11:52.712035 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712011    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kube-rbac-proxy"
Jun 15 06:11:52.712185 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712174    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kube-rbac-proxy"
Jun 15 06:11:52.712285 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712275    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerName="storage-initializer"
Jun 15 06:11:52.712357 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712349    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerName="storage-initializer"
Jun 15 06:11:52.712466 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712456    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerName="storage-initializer"
Jun 15 06:11:52.712536 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712527    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerName="storage-initializer"
Jun 15 06:11:52.712614 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712606    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="storage-initializer"
Jun 15 06:11:52.712680 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712672    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="storage-initializer"
Jun 15 06:11:52.712788 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712778    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container"
Jun 15 06:11:52.712864 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.712855    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container"
Jun 15 06:11:52.713072 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.713061    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kube-rbac-proxy"
Jun 15 06:11:52.713162 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.713153    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerName="storage-initializer"
Jun 15 06:11:52.713245 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.713237    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerName="kserve-container"
Jun 15 06:11:52.713594 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.713580    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9e63f1-cdeb-4e5b-9952-b48937fb1e74" containerName="storage-initializer"
Jun 15 06:11:52.717752 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.717715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.719914 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.719893    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-6fcbc-predictor-serving-cert\""
Jun 15 06:11:52.720014 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.719942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-6fcbc-kube-rbac-proxy-sar-config\""
Jun 15 06:11:52.723467 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.723441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"]
Jun 15 06:11:52.792122 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24r6b\" (UniqueName: \"kubernetes.io/projected/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-api-access-24r6b\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.792311 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.792311 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79511fbe-a197-4b18-add3-d6d5d81a2f32-proxy-tls\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.792311 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-cabundle-cert\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.792311 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79511fbe-a197-4b18-add3-d6d5d81a2f32-kserve-provision-location\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.792505 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792325    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:52.792505 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792354    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:52.792505 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792370    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:52.792505 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792386    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz5nh\" (UniqueName: \"kubernetes.io/projected/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-api-access-qz5nh\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:52.792505 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.792400    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f8fee9-da67-4069-a0e4-738cb7dc7a9c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:52.893502 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.893459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79511fbe-a197-4b18-add3-d6d5d81a2f32-kserve-provision-location\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.893712 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.893523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24r6b\" (UniqueName: \"kubernetes.io/projected/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-api-access-24r6b\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.893712 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.893546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.893712 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.893576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79511fbe-a197-4b18-add3-d6d5d81a2f32-proxy-tls\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.893712 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.893595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-cabundle-cert\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.893970 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.893951    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79511fbe-a197-4b18-add3-d6d5d81a2f32-kserve-provision-location\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.894273 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.894256    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-cabundle-cert\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.894324 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.894308    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.896055 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.896032    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79511fbe-a197-4b18-add3-d6d5d81a2f32-proxy-tls\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:52.901627 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:52.901607    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24r6b\" (UniqueName: \"kubernetes.io/projected/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-api-access-24r6b\") pod \"raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") " pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:53.030205 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.030105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:53.161057 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.161034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"]
Jun 15 06:11:53.163407 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:11:53.163377    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79511fbe_a197_4b18_add3_d6d5d81a2f32.slice/crio-df399d29f347053f31a16f640d09d5ec95ac13f1efa86cebe5e9601cbadaf463 WatchSource:0}: Error finding container df399d29f347053f31a16f640d09d5ec95ac13f1efa86cebe5e9601cbadaf463: Status 404 returned error can't find the container with id df399d29f347053f31a16f640d09d5ec95ac13f1efa86cebe5e9601cbadaf463
Jun 15 06:11:53.316295 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.316218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:11:53.368154 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.368103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:11:53.451956 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.451921    2579 generic.go:356] "Generic (PLEG): container finished" podID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" containerID="26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d" exitCode=0
Jun 15 06:11:53.452436 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.451995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"
Jun 15 06:11:53.452436 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.452010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerDied","Data":"26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d"}
Jun 15 06:11:53.452436 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.452077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc" event={"ID":"98f8fee9-da67-4069-a0e4-738cb7dc7a9c","Type":"ContainerDied","Data":"9a1908eb6a19b4c0c058f59fd0c0d13b10b21698e2598720429ab8849b5f2c6f"}
Jun 15 06:11:53.452436 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.452102    2579 scope.go:117] "RemoveContainer" containerID="68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83"
Jun 15 06:11:53.453898 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.453869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerStarted","Data":"62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a"}
Jun 15 06:11:53.454028 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.453945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerStarted","Data":"df399d29f347053f31a16f640d09d5ec95ac13f1efa86cebe5e9601cbadaf463"}
Jun 15 06:11:53.454100 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.453998    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerName="storage-initializer" containerID="cri-o://4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72" gracePeriod=30
Jun 15 06:11:53.460787 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.460761    2579 scope.go:117] "RemoveContainer" containerID="26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d"
Jun 15 06:11:53.468535 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.468515    2579 scope.go:117] "RemoveContainer" containerID="7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7"
Jun 15 06:11:53.477874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.477848    2579 scope.go:117] "RemoveContainer" containerID="68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83"
Jun 15 06:11:53.478180 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:53.478163    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83\": container with ID starting with 68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83 not found: ID does not exist" containerID="68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83"
Jun 15 06:11:53.478232 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.478189    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83"} err="failed to get container status \"68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83\": rpc error: code = NotFound desc = could not find container \"68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83\": container with ID starting with 68733945702b368043ce8ade641b226afb8457c6fcbe14ac328c9366825a9f83 not found: ID does not exist"
Jun 15 06:11:53.478232 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.478206    2579 scope.go:117] "RemoveContainer" containerID="26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d"
Jun 15 06:11:53.478483 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:53.478455    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d\": container with ID starting with 26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d not found: ID does not exist" containerID="26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d"
Jun 15 06:11:53.478548 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.478485    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d"} err="failed to get container status \"26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d\": rpc error: code = NotFound desc = could not find container \"26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d\": container with ID starting with 26cee89264264550e932ff029e6d868e8ab62e491f4e1582ad35af5f2008a19d not found: ID does not exist"
Jun 15 06:11:53.478548 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.478509    2579 scope.go:117] "RemoveContainer" containerID="7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7"
Jun 15 06:11:53.478827 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:53.478805    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7\": container with ID starting with 7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7 not found: ID does not exist" containerID="7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7"
Jun 15 06:11:53.478874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.478834    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7"} err="failed to get container status \"7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7\": rpc error: code = NotFound desc = could not find container \"7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7\": container with ID starting with 7e49d72819d1a6ed1e6d50bcc1f0cefd321ccb12005a194264df413f5d6377f7 not found: ID does not exist"
Jun 15 06:11:53.502616 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.502581    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"]
Jun 15 06:11:53.505740 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:53.505695    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f19eac-predictor-6ff95b59f6-9rsrc"]
Jun 15 06:11:54.158664 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.158631    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f8fee9-da67-4069-a0e4-738cb7dc7a9c" path="/var/lib/kubelet/pods/98f8fee9-da67-4069-a0e4-738cb7dc7a9c/volumes"
Jun 15 06:11:54.296043 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.296021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh_11a0ecfe-5f06-4878-a71f-8ac19e112e80/storage-initializer/1.log"
Jun 15 06:11:54.296412 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.296397    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh_11a0ecfe-5f06-4878-a71f-8ac19e112e80/storage-initializer/0.log"
Jun 15 06:11:54.298620 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.298604    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:54.402941 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.402904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-cabundle-cert\") pod \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") "
Jun 15 06:11:54.403149 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.402953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a0ecfe-5f06-4878-a71f-8ac19e112e80-proxy-tls\") pod \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") "
Jun 15 06:11:54.403149 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.403020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kserve-provision-location\") pod \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") "
Jun 15 06:11:54.403149 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.403047    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgzm5\" (UniqueName: \"kubernetes.io/projected/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-api-access-zgzm5\") pod \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") "
Jun 15 06:11:54.403149 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.403085    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-rbac-proxy-sar-config\") pod \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\" (UID: \"11a0ecfe-5f06-4878-a71f-8ac19e112e80\") "
Jun 15 06:11:54.403369 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.403259    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11a0ecfe-5f06-4878-a71f-8ac19e112e80" (UID: "11a0ecfe-5f06-4878-a71f-8ac19e112e80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:11:54.403369 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.403336    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11a0ecfe-5f06-4878-a71f-8ac19e112e80" (UID: "11a0ecfe-5f06-4878-a71f-8ac19e112e80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:54.403536 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.403508    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11a0ecfe-5f06-4878-a71f-8ac19e112e80" (UID: "11a0ecfe-5f06-4878-a71f-8ac19e112e80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:54.405156 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.405122    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a0ecfe-5f06-4878-a71f-8ac19e112e80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11a0ecfe-5f06-4878-a71f-8ac19e112e80" (UID: "11a0ecfe-5f06-4878-a71f-8ac19e112e80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:54.405222 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.405207    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-api-access-zgzm5" (OuterVolumeSpecName: "kube-api-access-zgzm5") pod "11a0ecfe-5f06-4878-a71f-8ac19e112e80" (UID: "11a0ecfe-5f06-4878-a71f-8ac19e112e80"). InnerVolumeSpecName "kube-api-access-zgzm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:54.458357 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.458329    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh_11a0ecfe-5f06-4878-a71f-8ac19e112e80/storage-initializer/1.log"
Jun 15 06:11:54.458774 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.458686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh_11a0ecfe-5f06-4878-a71f-8ac19e112e80/storage-initializer/0.log"
Jun 15 06:11:54.458774 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.458747    2579 generic.go:356] "Generic (PLEG): container finished" podID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerID="4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72" exitCode=1
Jun 15 06:11:54.458859 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.458808    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"
Jun 15 06:11:54.458859 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.458826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh" event={"ID":"11a0ecfe-5f06-4878-a71f-8ac19e112e80","Type":"ContainerDied","Data":"4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72"}
Jun 15 06:11:54.458945 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.458885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh" event={"ID":"11a0ecfe-5f06-4878-a71f-8ac19e112e80","Type":"ContainerDied","Data":"75c812927e2a3098cb98763d01e65e0ae500a06fc3d21040998bcbd2eb8cf225"}
Jun 15 06:11:54.458945 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.458921    2579 scope.go:117] "RemoveContainer" containerID="4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72"
Jun 15 06:11:54.468391 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.468373    2579 scope.go:117] "RemoveContainer" containerID="da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece"
Jun 15 06:11:54.475715 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.475696    2579 scope.go:117] "RemoveContainer" containerID="4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72"
Jun 15 06:11:54.475994 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:54.475972    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72\": container with ID starting with 4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72 not found: ID does not exist" containerID="4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72"
Jun 15 06:11:54.476033 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.476003    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72"} err="failed to get container status \"4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72\": rpc error: code = NotFound desc = could not find container \"4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72\": container with ID starting with 4a751ef1063b8beedba2bbe7e2fdda59ae72232be78c551299f33cedb577bf72 not found: ID does not exist"
Jun 15 06:11:54.476033 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.476029    2579 scope.go:117] "RemoveContainer" containerID="da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece"
Jun 15 06:11:54.476276 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:11:54.476261    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece\": container with ID starting with da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece not found: ID does not exist" containerID="da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece"
Jun 15 06:11:54.476311 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.476283    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece"} err="failed to get container status \"da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece\": rpc error: code = NotFound desc = could not find container \"da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece\": container with ID starting with da34fdfe3a462206859f8c48a190ee5d3bff62cc7150d077f9d6881aaec33ece not found: ID does not exist"
Jun 15 06:11:54.498597 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.498568    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"]
Jun 15 06:11:54.502263 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.502237    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4ed3d4-predictor-7b4f64db57-tt5xh"]
Jun 15 06:11:54.504268 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.504250    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:54.504347 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.504272    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a0ecfe-5f06-4878-a71f-8ac19e112e80-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:54.504347 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.504285    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:54.504347 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.504299    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgzm5\" (UniqueName: \"kubernetes.io/projected/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-api-access-zgzm5\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:54.504347 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:54.504312    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a0ecfe-5f06-4878-a71f-8ac19e112e80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:11:56.162849 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:56.160068    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" path="/var/lib/kubelet/pods/11a0ecfe-5f06-4878-a71f-8ac19e112e80/volumes"
Jun 15 06:11:57.472998 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:57.472907    2579 generic.go:356] "Generic (PLEG): container finished" podID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerID="62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a" exitCode=0
Jun 15 06:11:57.472998 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:57.472982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerDied","Data":"62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a"}
Jun 15 06:11:58.478874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:58.478840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerStarted","Data":"cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873"}
Jun 15 06:11:58.479261 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:58.478901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerStarted","Data":"65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1"}
Jun 15 06:11:58.479261 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:58.479142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:58.500380 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:58.500324    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podStartSLOduration=2.5672093240000002 podStartE2EDuration="6.500307946s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.933098622s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:11:58.498716207 +0000 UTC m=+1456.819633495" watchObservedRunningTime="2026-06-15 06:11:58.500307946 +0000 UTC m=+1456.821225232"
Jun 15 06:11:59.482412 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:59.482379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:11:59.483763 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:11:59.483736    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:12:00.485644 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:00.485551    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:12:05.490342 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:05.490311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:12:05.490917 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:05.490856    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:12:15.491748 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:15.491697    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:12:25.490877 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:25.490834    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:12:35.491626 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:35.491584    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:12:42.199081 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:42.199051    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 06:12:42.203002 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:42.202963    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 06:12:45.491162 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:45.491112    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:12:55.490850 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:12:55.490808    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:13:05.491458 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:05.491372    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:13:06.154797 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:06.154758    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:13:16.159224 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:16.159186    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:13:22.929342 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:22.929306    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"]
Jun 15 06:13:22.929755 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:22.929587    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" containerID="cri-o://65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1" gracePeriod=30
Jun 15 06:13:22.929755 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:22.929642    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kube-rbac-proxy" containerID="cri-o://cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873" gracePeriod=30
Jun 15 06:13:22.956518 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:22.956483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:13:23.054566 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.054532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:13:23.073121 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.073087    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"]
Jun 15 06:13:23.073361 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.073349    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerName="storage-initializer"
Jun 15 06:13:23.073409 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.073363    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerName="storage-initializer"
Jun 15 06:13:23.073444 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.073419    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerName="storage-initializer"
Jun 15 06:13:23.073444 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.073426    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerName="storage-initializer"
Jun 15 06:13:23.073506 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.073478    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerName="storage-initializer"
Jun 15 06:13:23.073506 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.073483    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11a0ecfe-5f06-4878-a71f-8ac19e112e80" containerName="storage-initializer"
Jun 15 06:13:23.076623 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.076602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.078652 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.078626    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-34378-predictor-serving-cert\""
Jun 15 06:13:23.078814 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.078761    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-34378-kube-rbac-proxy-sar-config\""
Jun 15 06:13:23.085447 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.085423    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"]
Jun 15 06:13:23.168021 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.167980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548f6155-f398-4ace-a589-722d96b7c487-proxy-tls\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.168021 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.168022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-cabundle-cert\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.168251 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.168053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548f6155-f398-4ace-a589-722d96b7c487-kserve-provision-location\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.168251 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.168116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.168251 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.168168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwc2w\" (UniqueName: \"kubernetes.io/projected/548f6155-f398-4ace-a589-722d96b7c487-kube-api-access-gwc2w\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.182937 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.182852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:13:23.269058 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwc2w\" (UniqueName: \"kubernetes.io/projected/548f6155-f398-4ace-a589-722d96b7c487-kube-api-access-gwc2w\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.269264 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548f6155-f398-4ace-a589-722d96b7c487-proxy-tls\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.269264 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-cabundle-cert\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.269264 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548f6155-f398-4ace-a589-722d96b7c487-kserve-provision-location\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.269264 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.269630 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269603    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548f6155-f398-4ace-a589-722d96b7c487-kserve-provision-location\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.269852 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-cabundle-cert\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.269952 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.269932    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.271890 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.271871    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548f6155-f398-4ace-a589-722d96b7c487-proxy-tls\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.278489 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.278463    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwc2w\" (UniqueName: \"kubernetes.io/projected/548f6155-f398-4ace-a589-722d96b7c487-kube-api-access-gwc2w\") pod \"raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.388116 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.388077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:23.522839 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.522810    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"]
Jun 15 06:13:23.525482 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:13:23.525455    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548f6155_f398_4ace_a589_722d96b7c487.slice/crio-162873f4b244ba4a8b5778d615c4e0ca3cb7139749efaeaee44556253aa2ed00 WatchSource:0}: Error finding container 162873f4b244ba4a8b5778d615c4e0ca3cb7139749efaeaee44556253aa2ed00: Status 404 returned error can't find the container with id 162873f4b244ba4a8b5778d615c4e0ca3cb7139749efaeaee44556253aa2ed00
Jun 15 06:13:23.793474 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.793382    2579 generic.go:356] "Generic (PLEG): container finished" podID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerID="cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873" exitCode=2
Jun 15 06:13:23.793474 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.793453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerDied","Data":"cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873"}
Jun 15 06:13:23.795034 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.795008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerStarted","Data":"9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f"}
Jun 15 06:13:23.795178 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:23.795058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerStarted","Data":"162873f4b244ba4a8b5778d615c4e0ca3cb7139749efaeaee44556253aa2ed00"}
Jun 15 06:13:25.485932 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:25.485885    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jun 15 06:13:26.154695 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:26.154660    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jun 15 06:13:27.811924 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:27.811887    2579 generic.go:356] "Generic (PLEG): container finished" podID="548f6155-f398-4ace-a589-722d96b7c487" containerID="9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f" exitCode=0
Jun 15 06:13:27.812327 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:27.811958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerDied","Data":"9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f"}
Jun 15 06:13:27.983179 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:27.983150    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:13:28.004914 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.004876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-rbac-proxy-sar-config\") pod \"79511fbe-a197-4b18-add3-d6d5d81a2f32\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") "
Jun 15 06:13:28.004914 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.004927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-cabundle-cert\") pod \"79511fbe-a197-4b18-add3-d6d5d81a2f32\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") "
Jun 15 06:13:28.005104 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.004961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24r6b\" (UniqueName: \"kubernetes.io/projected/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-api-access-24r6b\") pod \"79511fbe-a197-4b18-add3-d6d5d81a2f32\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") "
Jun 15 06:13:28.005104 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.004986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79511fbe-a197-4b18-add3-d6d5d81a2f32-proxy-tls\") pod \"79511fbe-a197-4b18-add3-d6d5d81a2f32\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") "
Jun 15 06:13:28.005104 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.005038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79511fbe-a197-4b18-add3-d6d5d81a2f32-kserve-provision-location\") pod \"79511fbe-a197-4b18-add3-d6d5d81a2f32\" (UID: \"79511fbe-a197-4b18-add3-d6d5d81a2f32\") "
Jun 15 06:13:28.005257 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.005221    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79511fbe-a197-4b18-add3-d6d5d81a2f32" (UID: "79511fbe-a197-4b18-add3-d6d5d81a2f32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:13:28.005318 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.005247    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79511fbe-a197-4b18-add3-d6d5d81a2f32" (UID: "79511fbe-a197-4b18-add3-d6d5d81a2f32"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:13:28.005456 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.005433    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79511fbe-a197-4b18-add3-d6d5d81a2f32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79511fbe-a197-4b18-add3-d6d5d81a2f32" (UID: "79511fbe-a197-4b18-add3-d6d5d81a2f32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:13:28.007837 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.007811    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-api-access-24r6b" (OuterVolumeSpecName: "kube-api-access-24r6b") pod "79511fbe-a197-4b18-add3-d6d5d81a2f32" (UID: "79511fbe-a197-4b18-add3-d6d5d81a2f32"). InnerVolumeSpecName "kube-api-access-24r6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:13:28.008133 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.007930    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79511fbe-a197-4b18-add3-d6d5d81a2f32-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79511fbe-a197-4b18-add3-d6d5d81a2f32" (UID: "79511fbe-a197-4b18-add3-d6d5d81a2f32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:13:28.105934 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.105896    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24r6b\" (UniqueName: \"kubernetes.io/projected/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-api-access-24r6b\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:13:28.105934 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.105934    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79511fbe-a197-4b18-add3-d6d5d81a2f32-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:13:28.106139 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.105944    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79511fbe-a197-4b18-add3-d6d5d81a2f32-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:13:28.106139 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.105953    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:13:28.106139 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.105963    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79511fbe-a197-4b18-add3-d6d5d81a2f32-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:13:28.817774 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.817708    2579 generic.go:356] "Generic (PLEG): container finished" podID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerID="65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1" exitCode=0
Jun 15 06:13:28.817774 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.817765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerDied","Data":"65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1"}
Jun 15 06:13:28.818322 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.817796    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"
Jun 15 06:13:28.818322 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.817824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7" event={"ID":"79511fbe-a197-4b18-add3-d6d5d81a2f32","Type":"ContainerDied","Data":"df399d29f347053f31a16f640d09d5ec95ac13f1efa86cebe5e9601cbadaf463"}
Jun 15 06:13:28.818322 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.817862    2579 scope.go:117] "RemoveContainer" containerID="cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873"
Jun 15 06:13:28.820457 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.820432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerStarted","Data":"55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b"}
Jun 15 06:13:28.820560 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.820495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerStarted","Data":"a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a"}
Jun 15 06:13:28.820898 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.820876    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:28.821022 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.820924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:28.822342 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.822311    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:13:28.825949 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.825933    2579 scope.go:117] "RemoveContainer" containerID="65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1"
Jun 15 06:13:28.833190 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.833171    2579 scope.go:117] "RemoveContainer" containerID="62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a"
Jun 15 06:13:28.840297 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.840282    2579 scope.go:117] "RemoveContainer" containerID="cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873"
Jun 15 06:13:28.840542 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:13:28.840525    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873\": container with ID starting with cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873 not found: ID does not exist" containerID="cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873"
Jun 15 06:13:28.840585 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.840551    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873"} err="failed to get container status \"cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873\": rpc error: code = NotFound desc = could not find container \"cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873\": container with ID starting with cec3ba41b1f7eea5bc8a705c03101dfe0cb36d8687923354e661895e98b9b873 not found: ID does not exist"
Jun 15 06:13:28.840585 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.840569    2579 scope.go:117] "RemoveContainer" containerID="65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1"
Jun 15 06:13:28.840800 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:13:28.840784    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1\": container with ID starting with 65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1 not found: ID does not exist" containerID="65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1"
Jun 15 06:13:28.840844 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.840804    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1"} err="failed to get container status \"65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1\": rpc error: code = NotFound desc = could not find container \"65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1\": container with ID starting with 65b34de8a84108237fcd05a9d7f33f89870acee8964bd236e1938a77e02c99d1 not found: ID does not exist"
Jun 15 06:13:28.840844 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.840819    2579 scope.go:117] "RemoveContainer" containerID="62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a"
Jun 15 06:13:28.841025 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:13:28.841011    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a\": container with ID starting with 62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a not found: ID does not exist" containerID="62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a"
Jun 15 06:13:28.841058 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.841030    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a"} err="failed to get container status \"62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a\": rpc error: code = NotFound desc = could not find container \"62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a\": container with ID starting with 62235958ff363d979f1a384ed258de6b2cae39d488488be0e278cc3ee052f07a not found: ID does not exist"
Jun 15 06:13:28.843272 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.843252    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"]
Jun 15 06:13:28.849756 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.849717    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-6fcbc-predictor-76cb49bc97-2x4x7"]
Jun 15 06:13:28.866326 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:28.866277    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podStartSLOduration=2.177775892 podStartE2EDuration="5.866263547s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688487655s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:13:28.864528469 +0000 UTC m=+1547.185445795" watchObservedRunningTime="2026-06-15 06:13:28.866263547 +0000 UTC m=+1547.187180833"
Jun 15 06:13:29.825931 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:29.825897    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:13:30.159464 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:30.159420    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" path="/var/lib/kubelet/pods/79511fbe-a197-4b18-add3-d6d5d81a2f32/volumes"
Jun 15 06:13:34.831380 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:34.831350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:13:34.831898 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:34.831872    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:13:44.832490 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:44.832449    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:13:54.831874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:13:54.831831    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:14:04.832107 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:04.832057    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:14:14.832271 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:14.832229    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:14:24.832053 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:24.832009    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:14:33.267313 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:33.267275    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:14:34.832377 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:34.832330    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:14:43.999719 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:43.999681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:14:44.832892 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:44.832866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:14:53.144745 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:53.144699    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"]
Jun 15 06:14:53.145235 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:53.145027    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" containerID="cri-o://a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a" gracePeriod=30
Jun 15 06:14:53.145235 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:53.145071    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kube-rbac-proxy" containerID="cri-o://55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b" gracePeriod=30
Jun 15 06:14:53.171837 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:53.171807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:14:54.131265 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:54.131231    2579 generic.go:356] "Generic (PLEG): container finished" podID="548f6155-f398-4ace-a589-722d96b7c487" containerID="55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b" exitCode=2
Jun 15 06:14:54.131452 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:54.131305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerDied","Data":"55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b"}
Jun 15 06:14:54.504969 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:54.504891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:14:54.603907 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:54.603870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:14:54.826217 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:54.826116    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jun 15 06:14:54.832409 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:54.832366    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jun 15 06:14:58.291036 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.291008    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:14:58.392252 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392154    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-cabundle-cert\") pod \"548f6155-f398-4ace-a589-722d96b7c487\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") "
Jun 15 06:14:58.392252 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548f6155-f398-4ace-a589-722d96b7c487-kserve-provision-location\") pod \"548f6155-f398-4ace-a589-722d96b7c487\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") "
Jun 15 06:14:58.392252 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-kube-rbac-proxy-sar-config\") pod \"548f6155-f398-4ace-a589-722d96b7c487\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") "
Jun 15 06:14:58.392543 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwc2w\" (UniqueName: \"kubernetes.io/projected/548f6155-f398-4ace-a589-722d96b7c487-kube-api-access-gwc2w\") pod \"548f6155-f398-4ace-a589-722d96b7c487\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") "
Jun 15 06:14:58.392543 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392309    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548f6155-f398-4ace-a589-722d96b7c487-proxy-tls\") pod \"548f6155-f398-4ace-a589-722d96b7c487\" (UID: \"548f6155-f398-4ace-a589-722d96b7c487\") "
Jun 15 06:14:58.392709 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392571    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "548f6155-f398-4ace-a589-722d96b7c487" (UID: "548f6155-f398-4ace-a589-722d96b7c487"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:14:58.392709 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392599    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548f6155-f398-4ace-a589-722d96b7c487-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "548f6155-f398-4ace-a589-722d96b7c487" (UID: "548f6155-f398-4ace-a589-722d96b7c487"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:14:58.392709 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.392643    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "548f6155-f398-4ace-a589-722d96b7c487" (UID: "548f6155-f398-4ace-a589-722d96b7c487"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:14:58.394439 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.394419    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548f6155-f398-4ace-a589-722d96b7c487-kube-api-access-gwc2w" (OuterVolumeSpecName: "kube-api-access-gwc2w") pod "548f6155-f398-4ace-a589-722d96b7c487" (UID: "548f6155-f398-4ace-a589-722d96b7c487"). InnerVolumeSpecName "kube-api-access-gwc2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:14:58.394527 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.394510    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548f6155-f398-4ace-a589-722d96b7c487-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "548f6155-f398-4ace-a589-722d96b7c487" (UID: "548f6155-f398-4ace-a589-722d96b7c487"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:14:58.493129 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.493077    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:14:58.493129 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.493119    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwc2w\" (UniqueName: \"kubernetes.io/projected/548f6155-f398-4ace-a589-722d96b7c487-kube-api-access-gwc2w\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:14:58.493129 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.493129    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548f6155-f398-4ace-a589-722d96b7c487-proxy-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:14:58.493129 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.493139    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548f6155-f398-4ace-a589-722d96b7c487-cabundle-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:14:58.493129 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:58.493147    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548f6155-f398-4ace-a589-722d96b7c487-kserve-provision-location\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Jun 15 06:14:59.155080 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.155041    2579 generic.go:356] "Generic (PLEG): container finished" podID="548f6155-f398-4ace-a589-722d96b7c487" containerID="a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a" exitCode=0
Jun 15 06:14:59.155296 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.155096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"
Jun 15 06:14:59.155296 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.155104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerDied","Data":"a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a"}
Jun 15 06:14:59.155296 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.155154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd" event={"ID":"548f6155-f398-4ace-a589-722d96b7c487","Type":"ContainerDied","Data":"162873f4b244ba4a8b5778d615c4e0ca3cb7139749efaeaee44556253aa2ed00"}
Jun 15 06:14:59.155296 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.155193    2579 scope.go:117] "RemoveContainer" containerID="55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b"
Jun 15 06:14:59.163869 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.163719    2579 scope.go:117] "RemoveContainer" containerID="a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a"
Jun 15 06:14:59.171529 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.171511    2579 scope.go:117] "RemoveContainer" containerID="9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f"
Jun 15 06:14:59.176562 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.176535    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"]
Jun 15 06:14:59.179259 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.179234    2579 scope.go:117] "RemoveContainer" containerID="55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b"
Jun 15 06:14:59.179548 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:14:59.179528    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b\": container with ID starting with 55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b not found: ID does not exist" containerID="55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b"
Jun 15 06:14:59.179609 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.179557    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b"} err="failed to get container status \"55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b\": rpc error: code = NotFound desc = could not find container \"55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b\": container with ID starting with 55e87ce2f38c27ce0e9820c6253533f7cb0bcfe46073ee69b8fc9fc731aed38b not found: ID does not exist"
Jun 15 06:14:59.179609 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.179576    2579 scope.go:117] "RemoveContainer" containerID="a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a"
Jun 15 06:14:59.179882 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:14:59.179864    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a\": container with ID starting with a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a not found: ID does not exist" containerID="a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a"
Jun 15 06:14:59.179941 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.179888    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a"} err="failed to get container status \"a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a\": rpc error: code = NotFound desc = could not find container \"a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a\": container with ID starting with a87855f7b7f1458b493dce56163ca0bfb897ae71da5d8c334b72d702e655114a not found: ID does not exist"
Jun 15 06:14:59.179941 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.179903    2579 scope.go:117] "RemoveContainer" containerID="9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f"
Jun 15 06:14:59.180144 ip-10-0-135-167 kubenswrapper[2579]: E0615 06:14:59.180127    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f\": container with ID starting with 9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f not found: ID does not exist" containerID="9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f"
Jun 15 06:14:59.180190 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.180150    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f"} err="failed to get container status \"9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f\": rpc error: code = NotFound desc = could not find container \"9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f\": container with ID starting with 9d03e1e68ea3baf96947a8c94e7b90dca7f0e4f5d05a7babf5dba0a5e89fa13f not found: ID does not exist"
Jun 15 06:14:59.180484 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:14:59.180465    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-34378-predictor-7679fff757-jfhxd"]
Jun 15 06:15:00.158970 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:00.158937    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548f6155-f398-4ace-a589-722d96b7c487" path="/var/lib/kubelet/pods/548f6155-f398-4ace-a589-722d96b7c487/volumes"
Jun 15 06:15:20.581318 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.581282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:15:20.717029 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.716991    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zskbr/must-gather-x555n"]
Jun 15 06:15:20.717256 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717244    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kube-rbac-proxy"
Jun 15 06:15:20.717306 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717257    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kube-rbac-proxy"
Jun 15 06:15:20.717306 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717268    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="storage-initializer"
Jun 15 06:15:20.717306 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717273    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="storage-initializer"
Jun 15 06:15:20.717306 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717282    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container"
Jun 15 06:15:20.717306 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717288    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container"
Jun 15 06:15:20.717306 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717299    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="storage-initializer"
Jun 15 06:15:20.717306 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717304    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="storage-initializer"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717314    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kube-rbac-proxy"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717319    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kube-rbac-proxy"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717326    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717331    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717376    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kube-rbac-proxy"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717382    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="548f6155-f398-4ace-a589-722d96b7c487" containerName="kserve-container"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717389    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kube-rbac-proxy"
Jun 15 06:15:20.717538 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.717395    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79511fbe-a197-4b18-add3-d6d5d81a2f32" containerName="kserve-container"
Jun 15 06:15:20.720101 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.720084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:20.722239 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.722214    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zskbr\"/\"kube-root-ca.crt\""
Jun 15 06:15:20.722937 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.722919    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zskbr\"/\"openshift-service-ca.crt\""
Jun 15 06:15:20.722994 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.722922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zskbr\"/\"default-dockercfg-gvfqk\""
Jun 15 06:15:20.727888 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.727863    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zskbr/must-gather-x555n"]
Jun 15 06:15:20.855621 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.855591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d093f58d-f3ad-4528-ba21-775c603f1c12-must-gather-output\") pod \"must-gather-x555n\" (UID: \"d093f58d-f3ad-4528-ba21-775c603f1c12\") " pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:20.855835 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.855630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k65s\" (UniqueName: \"kubernetes.io/projected/d093f58d-f3ad-4528-ba21-775c603f1c12-kube-api-access-5k65s\") pod \"must-gather-x555n\" (UID: \"d093f58d-f3ad-4528-ba21-775c603f1c12\") " pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:20.956326 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.956282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d093f58d-f3ad-4528-ba21-775c603f1c12-must-gather-output\") pod \"must-gather-x555n\" (UID: \"d093f58d-f3ad-4528-ba21-775c603f1c12\") " pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:20.956326 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.956331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k65s\" (UniqueName: \"kubernetes.io/projected/d093f58d-f3ad-4528-ba21-775c603f1c12-kube-api-access-5k65s\") pod \"must-gather-x555n\" (UID: \"d093f58d-f3ad-4528-ba21-775c603f1c12\") " pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:20.956647 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.956623    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d093f58d-f3ad-4528-ba21-775c603f1c12-must-gather-output\") pod \"must-gather-x555n\" (UID: \"d093f58d-f3ad-4528-ba21-775c603f1c12\") " pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:20.964652 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:20.964622    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k65s\" (UniqueName: \"kubernetes.io/projected/d093f58d-f3ad-4528-ba21-775c603f1c12-kube-api-access-5k65s\") pod \"must-gather-x555n\" (UID: \"d093f58d-f3ad-4528-ba21-775c603f1c12\") " pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:21.029721 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:21.029678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zskbr/must-gather-x555n"
Jun 15 06:15:21.159791 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:21.159766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zskbr/must-gather-x555n"]
Jun 15 06:15:21.162294 ip-10-0-135-167 kubenswrapper[2579]: W0615 06:15:21.162265    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd093f58d_f3ad_4528_ba21_775c603f1c12.slice/crio-a9eda66955375ff95913f896fe492f4d40d12e6450d70b25884c1761faa35901 WatchSource:0}: Error finding container a9eda66955375ff95913f896fe492f4d40d12e6450d70b25884c1761faa35901: Status 404 returned error can't find the container with id a9eda66955375ff95913f896fe492f4d40d12e6450d70b25884c1761faa35901
Jun 15 06:15:21.164511 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:21.164496    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:15:21.233349 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:21.233292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zskbr/must-gather-x555n" event={"ID":"d093f58d-f3ad-4528-ba21-775c603f1c12","Type":"ContainerStarted","Data":"a9eda66955375ff95913f896fe492f4d40d12e6450d70b25884c1761faa35901"}
Jun 15 06:15:21.632006 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:21.631970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:15:22.240495 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:22.240403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zskbr/must-gather-x555n" event={"ID":"d093f58d-f3ad-4528-ba21-775c603f1c12","Type":"ContainerStarted","Data":"dc9770e66824cc36dce1d7b70047c2e2d272a6dd0eb4817c84c162dbc953a3d5"}
Jun 15 06:15:22.240744 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:22.240701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zskbr/must-gather-x555n" event={"ID":"d093f58d-f3ad-4528-ba21-775c603f1c12","Type":"ContainerStarted","Data":"976707185d5da3d3bea93cd94d31d30e2fbd8d654579e60916cb01e8af5fbf98"}
Jun 15 06:15:22.257318 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:22.257240    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zskbr/must-gather-x555n" podStartSLOduration=1.523212693 podStartE2EDuration="2.257216951s" totalImagesPullingTime="734.004258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:15:22.254819318 +0000 UTC m=+1660.575737010" watchObservedRunningTime="2026-06-15 06:15:22.257216951 +0000 UTC m=+1660.578134235"
Jun 15 06:15:23.360760 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:23.360708    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7qgzl_369df3e2-42a9-4674-a0a9-3ed49caa5dc4/global-pull-secret-syncer/0.log"
Jun 15 06:15:23.459156 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:23.459120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log"
Jun 15 06:15:23.535587 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:23.535333    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-167.ec2.internal_9e3ac6eed3caa2ac33824ee71932a85f/haproxy/0.log"
Jun 15 06:15:26.150102 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:26.150015    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:15:26.205292 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:26.205260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c4zl_be6e0b77-c7f7-48a5-907f-9a5dd0ea60ad/konnectivity-agent/0.log" limit=1024
Jun 15 06:15:28.576215 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:28.576130    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n68pr_b35a58db-88d3-4da3-aee1-7b21a9cbf37a/node-exporter/0.log"
Jun 15 06:15:28.598213 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:28.598184    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n68pr_b35a58db-88d3-4da3-aee1-7b21a9cbf37a/kube-rbac-proxy/0.log"
Jun 15 06:15:28.621951 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:28.621925    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n68pr_b35a58db-88d3-4da3-aee1-7b21a9cbf37a/init-textfile/0.log"
Jun 15 06:15:30.453476 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:30.453446    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-867b76cb74-gvhjf_415c4a6f-c939-42f7-9c11-c1730085b403/networking-console-plugin/0.log"
Jun 15 06:15:31.709682 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.709640    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"]
Jun 15 06:15:31.715035 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.715010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.722063 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.722029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"]
Jun 15 06:15:31.856666 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.856623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-proc\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.856985 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.856965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-lib-modules\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.857134 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.857121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrrjc\" (UniqueName: \"kubernetes.io/projected/5e920418-a761-4081-b802-b0aaccbf0150-kube-api-access-wrrjc\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.857262 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.857249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-podres\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.857432 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.857407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-sys\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959011 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.958968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-proc\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-lib-modules\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959066    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-proc\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrrjc\" (UniqueName: \"kubernetes.io/projected/5e920418-a761-4081-b802-b0aaccbf0150-kube-api-access-wrrjc\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-podres\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959197 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-sys\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959414 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959222    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-lib-modules\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959414 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959273    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-sys\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.959414 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.959286    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e920418-a761-4081-b802-b0aaccbf0150-podres\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:31.967438 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:31.967338    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrrjc\" (UniqueName: \"kubernetes.io/projected/5e920418-a761-4081-b802-b0aaccbf0150-kube-api-access-wrrjc\") pod \"perf-node-gather-daemonset-48xj7\" (UID: \"5e920418-a761-4081-b802-b0aaccbf0150\") " pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:32.031795 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:32.031759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:32.194956 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:32.191951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"]
Jun 15 06:15:32.295315 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:32.295280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7" event={"ID":"5e920418-a761-4081-b802-b0aaccbf0150","Type":"ContainerStarted","Data":"783bfa5baafc5a3dda65e5f776a12e59fb43d0a900ea7a4f7be3e11342bfb28f"}
Jun 15 06:15:32.897955 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:32.897928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zpd25_5a981072-5ef0-49c2-a80d-1dc2dcc4488d/dns/0.log"
Jun 15 06:15:32.919462 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:32.919436    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zpd25_5a981072-5ef0-49c2-a80d-1dc2dcc4488d/kube-rbac-proxy/0.log"
Jun 15 06:15:32.994384 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:32.994355    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcp6n_d31e44b4-6f88-42da-b187-34d230a5ded0/dns-node-resolver/0.log"
Jun 15 06:15:33.303473 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:33.303379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7" event={"ID":"5e920418-a761-4081-b802-b0aaccbf0150","Type":"ContainerStarted","Data":"804551bf59901d672ef48c5cf1908907e1cdd5773c623c32f37285773c1b7c18"}
Jun 15 06:15:33.303645 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:33.303484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:33.317411 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:33.317353    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7" podStartSLOduration=2.317329334 podStartE2EDuration="2.317329334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 06:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:15:33.31618419 +0000 UTC m=+1671.637101477" watchObservedRunningTime="2026-06-15 06:15:33.317329334 +0000 UTC m=+1671.638246622"
Jun 15 06:15:33.466249 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:33.466220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2vs2q_11b70477-ceae-4d96-a015-b52d5659b1a6/node-ca/0.log"
Jun 15 06:15:34.770541 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:34.770509    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x8n46_1947ce46-e2e0-45af-ae40-738b891e5894/serve-healthcheck-canary/0.log"
Jun 15 06:15:35.144902 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:35.144865    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g7xhw_2fa8a952-44a1-48d5-90cf-cbd7966d5cac/kube-rbac-proxy/0.log"
Jun 15 06:15:35.168659 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:35.168623    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g7xhw_2fa8a952-44a1-48d5-90cf-cbd7966d5cac/exporter/0.log"
Jun 15 06:15:35.190129 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:35.190102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g7xhw_2fa8a952-44a1-48d5-90cf-cbd7966d5cac/extractor/0.log"
Jun 15 06:15:37.302186 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:37.302148    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6c7654bd99-d9hfx_817e29c6-b60a-48f8-97ad-72473abb589e/manager/0.log"
Jun 15 06:15:37.365523 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:37.365491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-jmhw5_6c5320aa-0600-481b-8e32-8c4cc5499b36/server/0.log"
Jun 15 06:15:37.457047 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:37.457007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-5fvck_312b2aac-d3e5-457a-aacc-4efbcb38f761/manager/0.log"
Jun 15 06:15:37.538583 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:37.538530    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cw8gh_72d11a8b-65a8-47a9-bc03-1bc0283f8499/s3-init/0.log"
Jun 15 06:15:37.562951 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:37.562871    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cw8gh_72d11a8b-65a8-47a9-bc03-1bc0283f8499/download-hf-model/0.log"
Jun 15 06:15:39.317261 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:39.317234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zskbr/perf-node-gather-daemonset-48xj7"
Jun 15 06:15:42.796186 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:42.796152    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-822r9_6c6082db-426d-49f5-8603-71bab34f199b/kube-multus-additional-cni-plugins/0.log"
Jun 15 06:15:42.822939 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:42.822895    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-822r9_6c6082db-426d-49f5-8603-71bab34f199b/egress-router-binary-copy/0.log"
Jun 15 06:15:42.845834 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:42.845809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-822r9_6c6082db-426d-49f5-8603-71bab34f199b/cni-plugins/0.log"
Jun 15 06:15:42.869111 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:42.869083    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-822r9_6c6082db-426d-49f5-8603-71bab34f199b/bond-cni-plugin/0.log"
Jun 15 06:15:42.894782 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:42.894747    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-822r9_6c6082db-426d-49f5-8603-71bab34f199b/routeoverride-cni/0.log"
Jun 15 06:15:42.918295 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:42.918262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-822r9_6c6082db-426d-49f5-8603-71bab34f199b/whereabouts-cni-bincopy/0.log"
Jun 15 06:15:42.938737 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:42.938702    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-822r9_6c6082db-426d-49f5-8603-71bab34f199b/whereabouts-cni/0.log"
Jun 15 06:15:43.301511 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:43.301480    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lfbdv_b232addf-4916-4b31-b4b3-fa4157ad304d/kube-multus/0.log"
Jun 15 06:15:43.470800 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:43.470709    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-smsbk_9a3746b9-06bd-418a-b727-fcc71c98d786/network-metrics-daemon/0.log"
Jun 15 06:15:43.490271 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:43.490241    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-smsbk_9a3746b9-06bd-418a-b727-fcc71c98d786/kube-rbac-proxy/0.log"
Jun 15 06:15:44.384474 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.384443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-controller/0.log"
Jun 15 06:15:44.410027 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.409999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/1.log"
Jun 15 06:15:44.410183 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.410151    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovn-acl-logging/0.log"
Jun 15 06:15:44.433174 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.433145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/kube-rbac-proxy-node/0.log"
Jun 15 06:15:44.452676 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.452643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/kube-rbac-proxy-ovn-metrics/0.log"
Jun 15 06:15:44.469752 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.469702    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/northd/0.log"
Jun 15 06:15:44.491917 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.491890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/nbdb/0.log"
Jun 15 06:15:44.511874 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.511841    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/sbdb/0.log"
Jun 15 06:15:44.553983 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:44.553953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnn2f_175a2c40-7bff-4021-a86a-524baa098f52/ovnkube-controller/0.log"
Jun 15 06:15:45.812770 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:45.812719    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-thbgq_915e92a0-de1d-46ad-ba10-bcab485dc2ad/network-check-target-container/0.log"
Jun 15 06:15:46.717849 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:46.717819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kjqsr_bda06406-4b48-4943-9447-16483d800bda/iptables-alerter/0.log"
Jun 15 06:15:47.373968 ip-10-0-135-167 kubenswrapper[2579]: I0615 06:15:47.373942    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-2mvd7_78c95a18-cd99-4d98-aa17-5334445a567d/tuned/0.log"