Jul 13 20:19:14.546275 ip-10-0-128-233 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 20:19:14.886674 ip-10-0-128-233 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:19:14.886674 ip-10-0-128-233 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 20:19:14.886674 ip-10-0-128-233 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:19:14.886674 ip-10-0-128-233 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 20:19:14.886674 ip-10-0-128-233 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:19:14.887367 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.887278    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 20:19:14.897451 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897303    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:14.897451 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897448    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:14.897451 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897454    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:14.897451 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897458    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897462    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897465    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897468    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897471    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897474    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897477    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897480    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897483    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897486    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897488    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897494    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897498    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897501    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897505    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897507    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897510    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897513    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897516    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:14.897690 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897519    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897521    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897524    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897528    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897530    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897533    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897536    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897538    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897541    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897545    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897548    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897551    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897554    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897556    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897559    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897562    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897565    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897568    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897570    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897573    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:14.898180 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897576    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897578    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897581    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897584    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897588    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897592    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897597    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897600    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897605    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897608    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897613    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897615    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897620    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897623    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897626    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897629    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897631    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897636    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897639    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:14.898684 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897643    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897646    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897650    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897653    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897657    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897659    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897662    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897666    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897669    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897672    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897674    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897676    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897679    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897682    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897687    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897689    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897691    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897694    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897696    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897699    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:14.899144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897702    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897706    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897710    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897713    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897716    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897718    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897721    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897723    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897726    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897728    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897731    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897733    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897736    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897738    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897741    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897743    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897746    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.897748    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898460    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898466    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:14.899750 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898470    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898472    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898475    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898478    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898481    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898483    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898486    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898488    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898491    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898494    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898497    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898499    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898501    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898504    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898506    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898509    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898512    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898514    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898517    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898519    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:14.900292 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898521    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898524    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898526    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898529    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898532    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898535    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898538    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898541    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898543    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898546    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898548    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898551    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898554    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898558    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898561    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898564    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898567    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898570    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898573    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:14.900817 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898575    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898578    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898581    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898584    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898587    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898589    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898591    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898594    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898596    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898599    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898602    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898604    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898607    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898609    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898611    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898614    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898617    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898619    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898622    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898625    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:14.901294 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898627    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898631    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898634    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898638    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898640    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898643    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898646    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898648    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898651    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898654    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898656    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898658    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898661    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898663    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898666    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898668    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898670    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898673    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898676    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:14.901792 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898680    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898682    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898685    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898689    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898691    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898694    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898697    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898699    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898702    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898704    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898711    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898713    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898716    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898718    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898721    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898723    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898726    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898728    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.898731    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899156    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:14.902259 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899162    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899166    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899169    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899172    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899174    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899177    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899180    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899183    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899186    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899188    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899191    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899194    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899196    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899200    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899204    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899207    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899210    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899212    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899215    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899217    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:14.902722 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899219    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899222    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899229    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899231    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899234    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899236    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899239    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899241    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899244    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899246    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899249    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899252    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899255    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899257    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899260    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899263    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899265    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899267    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899270    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:14.903210 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899272    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899275    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899277    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899280    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899282    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899285    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899288    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899290    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899295    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899299    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899302    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899305    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899308    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899310    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899313    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899318    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899321    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899323    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899326    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899329    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:14.903656 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899332    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899334    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899337    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899339    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899343    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899346    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899348    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899350    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899353    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899355    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899358    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899360    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899362    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899365    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899367    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899370    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899372    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899375    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899377    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899379    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:14.904178 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899382    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899384    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899387    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899389    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899392    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899394    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899396    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899401    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899404    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899406    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899409    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899411    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899414    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899416    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899418    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899421    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899423    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899427    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.899431    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899504    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 20:19:14.904675 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899511    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899516    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899521    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899525    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899529    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899535    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899541    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899545    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899548    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899552    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899555    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899559    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899561    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899564    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899567    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899570    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899573    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899577    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899580    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899586    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899589    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899592    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899601    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 20:19:14.905176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899603    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899606    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899609    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899612    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899615    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899618    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899621    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899624    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899627    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899631    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899634    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899636    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899639    2575 flags.go:64] FLAG: --enable-server="true"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899642    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899646    2575 flags.go:64] FLAG: --event-burst="100"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899649    2575 flags.go:64] FLAG: --event-qps="50"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899652    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899655    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899657    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899661    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899664    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899667    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899670    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899672    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899676    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 20:19:14.905705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899680    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899683    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899685    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899691    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899693    2575 flags.go:64] FLAG: --feature-gates=""
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899699    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899702    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899705    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899708    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899711    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899714    2575 flags.go:64] FLAG: --help="false"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899717    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-233.ec2.internal"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899720    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899723    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899726    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899729    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899732    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899735    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899738    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899741    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899743    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899746    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899749    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899752    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 13 20:19:14.906433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899754    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899757    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899760    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899763    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899766    2575 flags.go:64] FLAG: --lock-file=""
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899768    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899771    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899774    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899780    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899783    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899786    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899790    2575 flags.go:64] FLAG: --logging-format="text"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899793    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899796    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899799    2575 flags.go:64] FLAG: --manifest-url=""
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899801    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899805    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899808    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899812    2575 flags.go:64] FLAG: --max-pods="110"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899815    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899817    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899821    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899823    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899827    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899829    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 20:19:14.907027 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899832    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899839    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899842    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899844    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899847    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899850    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899855    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899858    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899861    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899864    2575 flags.go:64] FLAG: --port="10250"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899867    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899869    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d857a5a44c92d61e"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899873    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899876    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899879    2575 flags.go:64] FLAG: --register-node="true"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899881    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899885    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899888    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899891    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899894    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899898    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899901    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899904    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899907    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899910    2575 flags.go:64] FLAG: --runonce="false"
Jul 13 20:19:14.907619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899913    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899916    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899919    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899921    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899924    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899927    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899930    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899933    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899936    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899939    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899942    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899945    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899948    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899951    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899953    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899958    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899961    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899964    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899967    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899970    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899973    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899976    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899979    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.899982    2575 flags.go:64] FLAG: --v="2"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.900001    2575 flags.go:64] FLAG: --version="false"
Jul 13 20:19:14.908216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.900008    2575 flags.go:64] FLAG: --vmodule=""
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.900011    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.900014    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900124    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900128    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900131    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900134    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900137    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900140    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900142    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900145    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900148    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900150    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900153    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900155    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900157    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900160    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900163    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900165    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900168    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:14.908825 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900170    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900172    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900175    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900177    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900179    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900182    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900184    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900186    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900189    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900191    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900194    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900196    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900198    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900201    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900203    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900206    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900208    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900211    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900213    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900215    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:14.909316 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900218    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900220    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900222    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900225    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900227    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900229    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900231    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900234    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900236    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900239    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900241    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900243    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900246    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900248    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900250    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900254    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900258    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900262    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900265    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:14.909818 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900267    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900269    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900272    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900274    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900277    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900281    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900283    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900286    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900289    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900291    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900294    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900296    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900299    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900301    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900304    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900306    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900309    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900311    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900314    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:14.910293 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900316    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900319    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900321    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900325    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900328    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900331    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900334    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900336    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900339    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900342    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900346    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900350    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900353    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900355    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900358    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900361    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900363    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900366    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900371    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:14.910751 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900373    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900376    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900379    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900382    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.900384    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.901016    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.907720    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.907735    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.907749    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.907754    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907811    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907816    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907819    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907822    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907825    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:14.911213 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907827    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907830    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907833    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907836    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907838    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907841    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907843    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907846    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907848    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907852    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907855    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907859    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907862    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907865    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907867    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907870    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907873    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907875    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907878    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907881    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:14.911586 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907884    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907887    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907889    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907892    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907895    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907897    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907900    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907903    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907905    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907908    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907910    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907913    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907915    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907918    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907920    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907923    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907925    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907928    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907930    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907932    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:14.912073 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907935    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907938    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907940    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907943    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907945    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907947    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907950    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907952    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907957    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907960    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907964    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907967    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907975    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907978    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907981    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.907984    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908007    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908010    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908013    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:14.912558 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908015    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908018    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908020    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908023    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908025    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908028    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908030    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908033    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908035    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908038    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908041    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908045    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908047    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908050    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908052    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908054    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908057    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908060    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908062    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:14.913082 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908064    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908067    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908070    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908072    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908075    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908077    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908080    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908084    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908086    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908088    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908092    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908094    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908097    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908099    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908102    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:14.913526 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908105    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.908110    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908242    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908248    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908251    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908254    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908257    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908260    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908263    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908267    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908269    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908272    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908275    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908277    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908280    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:14.913880 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908282    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908285    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908288    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908290    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908293    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908296    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908298    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908301    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908303    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908306    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908309    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908311    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908314    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908316    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908319    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908321    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908324    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908326    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908328    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908331    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:14.914257 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908333    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908336    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908338    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908341    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908343    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908345    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908348    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908350    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908353    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908355    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908358    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908360    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908362    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908365    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908367    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908370    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908372    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908375    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908377    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:14.914728 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908379    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908382    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908384    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908386    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908390    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908394    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908397    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908399    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908402    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908404    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908407    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908409    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908411    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908414    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908416    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908419    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908421    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908424    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908426    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908428    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:14.915249 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908431    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908433    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908435    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908438    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908440    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908443    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908445    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908447    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908449    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908452    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908454    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908457    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908459    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908461    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908464    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908466    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908468    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908471    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908474    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908478    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:14.915730 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908481    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908485    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908488    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908491    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908494    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908497    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:14.908500    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.908504    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.908607    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.911345    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.912541    2575 server.go:1019] "Starting client certificate rotation"
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.912638    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.912675    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 20:19:14.916231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.914756    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 20:19:14.916546 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.916209    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 20:19:14.925930 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.925907    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 13 20:19:14.928083 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.928067    2575 log.go:25] "Validated CRI v1 image API"
Jul 13 20:19:14.928842 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.928827    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 20:19:14.933478 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.933461    2575 fs.go:135] Filesystem UUIDs: map[3144ec48-f9c8-4ff8-9e56-892221dc1145:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b33ec8ba-a53b-49d7-81e5-1a1e28ceed75:/dev/nvme0n1p3]
Jul 13 20:19:14.933526 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.933478    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 20:19:14.938267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.938160    2575 manager.go:216] Machine: {Timestamp:2026-07-13 20:19:14.937010894 +0000 UTC m=+0.289619727 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116049 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24cca9271bfce8e35ccc894824a3a4 SystemUUID:ec24cca9-271b-fce8-e35c-cc894824a3a4 BootID:5b8b2b1c-8f48-481b-bf84-c33f56136af2 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:46:26:36:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:46:26:36:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:03:39:44:1d:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 20:19:14.938267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.938266    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 20:19:14.938374 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.938340    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 20:19:14.940979 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.940953    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 20:19:14.941160 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.940983    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 20:19:14.941663 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.941653    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 20:19:14.941698 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.941667    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 20:19:14.941698 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.941674    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 20:19:14.942246 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.942235    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 20:19:14.943568 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.943558    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 20:19:14.947218 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947206    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 20:19:14.947345 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947335    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 20:19:14.947509 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947501    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 20:19:14.947540 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947513    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 20:19:14.947540 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947528    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 20:19:14.947540 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947537    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 13 20:19:14.947614 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947546    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 20:19:14.947947 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947927    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 20:19:14.948040 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.947954    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 20:19:14.949969 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.949940    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 20:19:14.950059 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.950008    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 20:19:14.951864 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.951848    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 20:19:14.951905 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.951881    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 20:19:14.952079 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952067    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 20:19:14.952142 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952087    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 20:19:14.952142 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952095    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 20:19:14.952142 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952100    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 20:19:14.952142 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952109    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 20:19:14.952142 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952116    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 20:19:14.952441 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952432    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 20:19:14.952476 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952441    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 20:19:14.952476 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952449    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 20:19:14.952476 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952454    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 20:19:14.952476 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952463    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 20:19:14.952476 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952471    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 20:19:14.952657 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952508    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 20:19:14.952657 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.952514    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 20:19:14.955751 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.955739    2575 server.go:1268] "Started kubelet"
Jul 13 20:19:14.955973 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.955941    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 20:19:14.956162 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.956009    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 20:19:14.956289 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.956276    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 20:19:14.957213 ip-10-0-128-233 systemd[1]: Started Kubernetes Kubelet.
Jul 13 20:19:14.957546 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.957511    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 20:19:14.958568 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.958554    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 20:19:14.960161 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.960126    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 20:19:14.960306 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:14.960278    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 20:19:14.960371 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:14.960352    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 20:19:14.963251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.963232    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 20:19:14.963344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.963236    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 20:19:14.963480 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:14.963458    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:14.964044 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.964024    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 20:19:14.964044 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.964045    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 20:19:14.964175 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:14.960449    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-233.ec2.internal.18c1f2dc31af19a5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-233.ec2.internal,UID:ip-10-0-128-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-233.ec2.internal,},FirstTimestamp:2026-07-13 20:19:14.955712933 +0000 UTC m=+0.308321767,LastTimestamp:2026-07-13 20:19:14.955712933 +0000 UTC m=+0.308321767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-233.ec2.internal,}"
Jul 13 20:19:14.964175 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.964159    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 20:19:14.964175 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.964169    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 20:19:14.964175 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.964170    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 20:19:14.966131 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.966107    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 20:19:14.966131 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.966133    2575 factory.go:55] Registering systemd factory
Jul 13 20:19:14.966285 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.966141    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 13 20:19:14.966753 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:14.966507    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 20:19:14.966753 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:14.966666    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 20:19:14.966974 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.966958    2575 factory.go:153] Registering CRI-O factory
Jul 13 20:19:14.966974 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.966975    2575 factory.go:223] Registration of the crio container factory successfully
Jul 13 20:19:14.967102 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.967012    2575 factory.go:103] Registering Raw factory
Jul 13 20:19:14.967102 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.967025    2575 manager.go:1196] Started watching for new ooms in manager
Jul 13 20:19:14.967102 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:14.967003    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 20:19:14.967466 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.967453    2575 manager.go:349] Starting recovery of all containers
Jul 13 20:19:14.978856 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.978705    2575 manager.go:354] Recovery completed
Jul 13 20:19:14.980202 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.980180    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r276c"
Jul 13 20:19:14.982927 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.982915    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:14.984819 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.984805    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:14.984875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.984831    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:14.984875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.984841    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:14.985390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.985376    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 20:19:14.985390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.985387    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 20:19:14.985504 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.985401    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 20:19:14.985823 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.985811    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:14.986513 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.986499    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:14.986569 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.986525    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:14.986569 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.986536    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:14.987457 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.987445    2575 policy_none.go:49] "None policy: Start"
Jul 13 20:19:14.987505 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.987460    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 20:19:14.987505 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.987471    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 20:19:14.989188 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.989171    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r276c"
Jul 13 20:19:14.989400 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:14.989389    2575 policy_none.go:47] "Start"
Jul 13 20:19:15.019527 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.019514    2575 manager.go:337] "Starting Device Plugin manager"
Jul 13 20:19:15.019603 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.019544    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 20:19:15.019603 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.019554    2575 server.go:87] "Starting device plugin registration server"
Jul 13 20:19:15.019794 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.019781    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 20:19:15.019832 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.019799    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 20:19:15.019912 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.019895    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 20:19:15.020011 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.019981    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 20:19:15.020011 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.020011    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 20:19:15.021322 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.021299    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 20:19:15.021418 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.021346    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.120468 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.120439    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:15.122523 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.121695    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:15.122523 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.121723    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:15.122523 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.121739    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:15.122523 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.121771    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.122523 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.122350    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 20:19:15.123756 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.123742    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 20:19:15.123805 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.123775    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 20:19:15.123805 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.123797    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 20:19:15.123892 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.123825    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 20:19:15.126517 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.126495    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:15.130802 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.130782    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.130883 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.130806    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-233.ec2.internal\": node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.178304 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.178225    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.225216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.224560    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal"]
Jul 13 20:19:15.225336 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.225259    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:15.226208 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.226192    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:15.226297 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.226222    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:15.226297 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.226233    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:15.228525 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.228513    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:15.228651 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.228632    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.228687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.228683    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:15.229337 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.229320    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:15.229337 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.229327    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:15.229485 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.229346    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:15.229485 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.229349    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:15.229485 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.229358    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:15.229485 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.229364    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:15.231463 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.231450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.231512 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.231480    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:15.232112 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.232098    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:15.232184 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.232117    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:15.232184 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.232126    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:15.258511 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.258484    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-233.ec2.internal\" not found" node="ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.262972 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.262955    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-233.ec2.internal\" not found" node="ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.265390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.265374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/696cfbde432fcad9786b56905bc126bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal\" (UID: \"696cfbde432fcad9786b56905bc126bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.265447 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.265397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/696cfbde432fcad9786b56905bc126bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal\" (UID: \"696cfbde432fcad9786b56905bc126bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.265447 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.265415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3a758c83d40d2b976cf492c86764c97-config\") pod \"kube-apiserver-proxy-ip-10-0-128-233.ec2.internal\" (UID: \"e3a758c83d40d2b976cf492c86764c97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.278599 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.278582    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.365711 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.365683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3a758c83d40d2b976cf492c86764c97-config\") pod \"kube-apiserver-proxy-ip-10-0-128-233.ec2.internal\" (UID: \"e3a758c83d40d2b976cf492c86764c97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.365871 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.365716    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/696cfbde432fcad9786b56905bc126bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal\" (UID: \"696cfbde432fcad9786b56905bc126bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.365871 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.365738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/696cfbde432fcad9786b56905bc126bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal\" (UID: \"696cfbde432fcad9786b56905bc126bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.365871 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.365782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/696cfbde432fcad9786b56905bc126bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal\" (UID: \"696cfbde432fcad9786b56905bc126bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.365871 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.365787    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3a758c83d40d2b976cf492c86764c97-config\") pod \"kube-apiserver-proxy-ip-10-0-128-233.ec2.internal\" (UID: \"e3a758c83d40d2b976cf492c86764c97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.365871 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.365796    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/696cfbde432fcad9786b56905bc126bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal\" (UID: \"696cfbde432fcad9786b56905bc126bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.379395 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.379361    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.480157 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.480130    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.560313 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.560281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.566236 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.566217    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:15.581002 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.580966    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.681486 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.681447    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.782061 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.781970    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.812964 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.812942    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:15.883040 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.883011    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.886415 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.886399    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:15.912212 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.912191    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 20:19:15.912687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.912330    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 20:19:15.912687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.912380    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 20:19:15.912687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.912380    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 20:19:15.983148 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:15.983118    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-233.ec2.internal\" not found"
Jul 13 20:19:15.993396 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.993355    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 20:14:15 +0000 UTC" deadline="2028-01-23 15:45:02.702402075 +0000 UTC"
Jul 13 20:19:15.993396 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:15.993393    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13411h25m46.709014269s"
Jul 13 20:19:16.059820 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.059748    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:16.063975 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.063949    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:16.076389 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.076365    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 20:19:16.077447 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.077431    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal"
Jul 13 20:19:16.092709 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.092690    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 20:19:16.102003 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:16.101964    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a758c83d40d2b976cf492c86764c97.slice/crio-dfc512af25ac9224227391f188b81a0c3f285237380ca9a5bb386c501f0cebe2 WatchSource:0}: Error finding container dfc512af25ac9224227391f188b81a0c3f285237380ca9a5bb386c501f0cebe2: Status 404 returned error can't find the container with id dfc512af25ac9224227391f188b81a0c3f285237380ca9a5bb386c501f0cebe2
Jul 13 20:19:16.102228 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:16.102206    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696cfbde432fcad9786b56905bc126bd.slice/crio-f6a1c5b818174a64ec3008e95bda65c3f80f79fac0c9afacdd71ba6c37748529 WatchSource:0}: Error finding container f6a1c5b818174a64ec3008e95bda65c3f80f79fac0c9afacdd71ba6c37748529: Status 404 returned error can't find the container with id f6a1c5b818174a64ec3008e95bda65c3f80f79fac0c9afacdd71ba6c37748529
Jul 13 20:19:16.105361 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.105343    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:19:16.126806 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.126752    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal" event={"ID":"e3a758c83d40d2b976cf492c86764c97","Type":"ContainerStarted","Data":"dfc512af25ac9224227391f188b81a0c3f285237380ca9a5bb386c501f0cebe2"}
Jul 13 20:19:16.127812 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.127789    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal" event={"ID":"696cfbde432fcad9786b56905bc126bd","Type":"ContainerStarted","Data":"f6a1c5b818174a64ec3008e95bda65c3f80f79fac0c9afacdd71ba6c37748529"}
Jul 13 20:19:16.726581 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.726547    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:16.949528 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.949484    2575 apiserver.go:52] "Watching apiserver"
Jul 13 20:19:16.960092 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.959773    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 20:19:16.962047 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.961955    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-hc99x","kube-system/global-pull-secret-syncer-hqxqb","openshift-multus/multus-v25wz","openshift-multus/network-metrics-daemon-nrn7d","openshift-network-diagnostics/network-check-target-r557g","openshift-network-operator/iptables-alerter-9tj22","openshift-ovn-kubernetes/ovnkube-node-cwl74","kube-system/konnectivity-agent-jnrr6","kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942","openshift-cluster-node-tuning-operator/tuned-lxj57","openshift-dns/node-resolver-gzfvp","openshift-image-registry/node-ca-68pm9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal"]
Jul 13 20:19:16.964195 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.964119    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 20:19:16.965118 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.965099    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.967556 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.967538    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:16.970104 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.970077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v25wz"
Jul 13 20:19:16.972560 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.972540    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:16.972659 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:16.972613    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:16.972871 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.972843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-run-netns\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973013 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.972882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-etc-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973013 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.972906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973013 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.972940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovnkube-script-lib\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973013 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.972971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-systemd\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973013 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.972998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovn-node-metrics-cert\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-host\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-serviceca\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-kubelet\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-node-log\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-log-socket\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovnkube-config\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-env-overrides\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-systemd-units\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973262    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-var-lib-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-ovn\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-cni-bin\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-cni-netd\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973368    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973405    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hqqp\" (UniqueName: \"kubernetes.io/projected/1de3016b-df9e-4924-8f6b-5f42bbc831d8-kube-api-access-6hqqp\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vxmk\" (UniqueName: \"kubernetes.io/projected/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-kube-api-access-7vxmk\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:16.973676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.973454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-slash\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:16.975282 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.975263    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:16.975410 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:16.975331    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:16.983671 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.983607    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 20:19:16.983906 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.983891    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 20:19:16.984040 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.984000    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 20:19:16.984261 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.984243    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:16.985514 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.985492    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 20:19:16.985768 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.985750    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 20:19:16.986111 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.986096    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x2nks\""
Jul 13 20:19:16.986198 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.986187    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 20:19:16.987295 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.987276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:16.991565 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.991517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:16.994927 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.994807    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:16.994927 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:16.994879    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:16.997977 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.997960    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:16.998758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.998239    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 20:19:16.998758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.998424    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 20:19:16.998758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.998441    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q47vc\""
Jul 13 20:19:16.998758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.998511    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 20:19:16.998758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.998616    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 20:19:16.999079 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:16.999057    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 20:19:17.003694 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.003675    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.006171 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.006141    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.014089 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.013740    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 20:19:17.017204 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.016277    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9xwq\""
Jul 13 20:19:17.017204 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.016524    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 20:19:17.017204 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.016628    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 20:19:17.017204 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.016757    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 20:19:17.017564 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.017545    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 20:19:17.017798 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.017780    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 20:19:17.017866 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.017836    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 20:19:17.018003 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.017984    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 20:19:17.018129 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.018109    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 20:19:17.018174 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.018122    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5k5qk\""
Jul 13 20:19:17.018261 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.018237    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 20:19:17.018446 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.018420    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 20:19:17.018559 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.018513    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 20:19:17.019547 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.019524    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 20:19:17.019647 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.019625    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rckvq\""
Jul 13 20:19:17.019794 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.019770    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t7d52\""
Jul 13 20:19:17.019859 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.019842    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tbzbj\""
Jul 13 20:19:17.020217 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.020120    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 20:19:17.020217 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.020153    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mp72q\""
Jul 13 20:19:17.020409 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.020389    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nxspx\""
Jul 13 20:19:17.020486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.020441    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 20:19:17.030132 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.030109    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 20:19:17.034252 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.034230    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 20:19:17.038158 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.038137    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpt9l"
Jul 13 20:19:17.048280 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.048257    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gpt9l"
Jul 13 20:19:17.065617 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.065599    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 20:19:17.074001 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.073975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqqp\" (UniqueName: \"kubernetes.io/projected/1de3016b-df9e-4924-8f6b-5f42bbc831d8-kube-api-access-6hqqp\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074111 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.074111 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-kubernetes\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074111 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/636333a8-2370-4e1f-acc9-4ad49e62fcfc-dbus\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.074111 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.074326 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-run\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074326 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-systemd\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074326 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074326 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-serviceca\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:17.074326 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-conf-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.074326 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074293    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-systemd\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqw6\" (UniqueName: \"kubernetes.io/projected/baf87b37-94c8-4788-b01b-bd97a1ec5c63-kube-api-access-wjqw6\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56r6n\" (UniqueName: \"kubernetes.io/projected/7a052240-fd7b-4102-9bb4-b2b9506f9961-kube-api-access-56r6n\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-var-lib-kubelet\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-cni-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074483    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-netns\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-registration-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysctl-conf\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-lib-modules\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-host\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-system-cni-dir\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-sys-fs\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysconfig\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9111d6-4502-4d2d-8594-89f4281983f3-tmp\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bgc\" (UniqueName: \"kubernetes.io/projected/6b9111d6-4502-4d2d-8594-89f4281983f3-kube-api-access-75bgc\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-slash\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-serviceca\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-etc-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074841    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-slash\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-k8s-cni-cncf-io\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074895    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-etc-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-kubelet\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngdk\" (UniqueName: \"kubernetes.io/projected/7b192fbb-a258-4033-881f-1365ace9d445-kube-api-access-wngdk\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.074984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.074960    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovn-node-metrics-cert\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-os-release\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-cni-bin\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-hostroot\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-socket-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysctl-d\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-log-socket\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075276    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-log-socket\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075268    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075297    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5gj\" (UniqueName: \"kubernetes.io/projected/28f13592-025b-493b-a761-aab0e1bfda3c-kube-api-access-7z5gj\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-kube-api-access-8hlvb\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075344    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf87b37-94c8-4788-b01b-bd97a1ec5c63-cni-binary-copy\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-cni-multus\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-daemon-config\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-tuned\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075441    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-multus-certs\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.075573 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-systemd\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075507    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-sys\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-run-netns\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sf76\" (UniqueName: \"kubernetes.io/projected/1347ce06-e218-4be5-9dd2-5db94f75ff35-kube-api-access-7sf76\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075588    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-run-netns\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bbc94-91c8-4a0d-bc00-ea9be56778d2-konnectivity-ca\") pod \"konnectivity-agent-jnrr6\" (UID: \"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2\") " pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075599    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/636333a8-2370-4e1f-acc9-4ad49e62fcfc-kubelet-config\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovnkube-config\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-env-overrides\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-os-release\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-socket-dir-parent\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-cni-bin\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075861    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-cni-bin\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-cni-netd\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.075978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-cni-netd\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vxmk\" (UniqueName: \"kubernetes.io/projected/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-kube-api-access-7vxmk\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076106    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-tmp-dir\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-etc-kubernetes\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-modprobe-d\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-env-overrides\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovnkube-script-lib\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076241    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovnkube-config\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-host\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076616    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovnkube-script-lib\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076852    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-host\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076899    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-cnibin\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-hosts-file\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-cnibin\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.076996 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a052240-fd7b-4102-9bb4-b2b9506f9961-host-slash\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.076998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-kubelet\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-node-log\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-kubelet\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-system-cni-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a052240-fd7b-4102-9bb4-b2b9506f9961-iptables-alerter-script\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077165    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f4bbc94-91c8-4a0d-bc00-ea9be56778d2-agent-certs\") pod \"konnectivity-agent-jnrr6\" (UID: \"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2\") " pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-device-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-systemd-units\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-var-lib-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077264    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-systemd-units\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-ovn\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077383    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077404    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-var-lib-openvswitch\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.077875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-node-log\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.078649 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.077488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1de3016b-df9e-4924-8f6b-5f42bbc831d8-run-ovn\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.078692 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.078667    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1de3016b-df9e-4924-8f6b-5f42bbc831d8-ovn-node-metrics-cert\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.095121 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.093554    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqqp\" (UniqueName: \"kubernetes.io/projected/1de3016b-df9e-4924-8f6b-5f42bbc831d8-kube-api-access-6hqqp\") pod \"ovnkube-node-cwl74\" (UID: \"1de3016b-df9e-4924-8f6b-5f42bbc831d8\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.095121 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.093997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vxmk\" (UniqueName: \"kubernetes.io/projected/6498e489-3b9d-4ca6-8191-b2cd12b9c37a-kube-api-access-7vxmk\") pod \"node-ca-68pm9\" (UID: \"6498e489-3b9d-4ca6-8191-b2cd12b9c37a\") " pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:17.177861 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.177828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqw6\" (UniqueName: \"kubernetes.io/projected/baf87b37-94c8-4788-b01b-bd97a1ec5c63-kube-api-access-wjqw6\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178067 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.177869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56r6n\" (UniqueName: \"kubernetes.io/projected/7a052240-fd7b-4102-9bb4-b2b9506f9961-kube-api-access-56r6n\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.178067 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.177893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-var-lib-kubelet\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178067 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.177919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-cni-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178067 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.177990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-cni-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178067 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178066    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-var-lib-kubelet\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-netns\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-registration-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysctl-conf\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178214    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-netns\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178257    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-registration-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-lib-modules\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178322    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysctl-conf\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-host\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-system-cni-dir\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.178383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178376    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-host\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-sys-fs\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysconfig\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-system-cni-dir\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9111d6-4502-4d2d-8594-89f4281983f3-tmp\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-lib-modules\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bgc\" (UniqueName: \"kubernetes.io/projected/6b9111d6-4502-4d2d-8594-89f4281983f3-kube-api-access-75bgc\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-k8s-cni-cncf-io\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178533    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-sys-fs\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-kubelet\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngdk\" (UniqueName: \"kubernetes.io/projected/7b192fbb-a258-4033-881f-1365ace9d445-kube-api-access-wngdk\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysconfig\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-os-release\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-cni-bin\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-hostroot\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-socket-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.178773 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysctl-d\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5gj\" (UniqueName: \"kubernetes.io/projected/28f13592-025b-493b-a761-aab0e1bfda3c-kube-api-access-7z5gj\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178771    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-kube-api-access-8hlvb\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf87b37-94c8-4788-b01b-bd97a1ec5c63-cni-binary-copy\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-kubelet\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-cni-multus\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-daemon-config\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-tuned\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178931    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-multus-certs\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.178979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-systemd\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-sys\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179051    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sf76\" (UniqueName: \"kubernetes.io/projected/1347ce06-e218-4be5-9dd2-5db94f75ff35-kube-api-access-7sf76\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bbc94-91c8-4a0d-bc00-ea9be56778d2-konnectivity-ca\") pod \"konnectivity-agent-jnrr6\" (UID: \"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2\") " pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179063    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-os-release\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179126    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-k8s-cni-cncf-io\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.179818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/636333a8-2370-4e1f-acc9-4ad49e62fcfc-kubelet-config\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179175    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-cni-multus\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-os-release\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179218    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-var-lib-cni-bin\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-socket-dir-parent\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-tmp-dir\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-socket-dir-parent\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-etc-kubernetes\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-hostroot\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-socket-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-sysctl-d\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.179721    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-etc-kubernetes\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-host-run-multus-certs\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180287    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bbc94-91c8-4a0d-bc00-ea9be56778d2-konnectivity-ca\") pod \"konnectivity-agent-jnrr6\" (UID: \"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2\") " pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180292    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-systemd\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180348    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-sys\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180361    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.180680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180425    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/636333a8-2370-4e1f-acc9-4ad49e62fcfc-kubelet-config\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180654    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-daemon-config\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180754    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-os-release\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-modprobe-d\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf87b37-94c8-4788-b01b-bd97a1ec5c63-cni-binary-copy\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.180999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-cnibin\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-cnibin\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181044    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-tmp-dir\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-hosts-file\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181095    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-cnibin\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a052240-fd7b-4102-9bb4-b2b9506f9961-host-slash\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181130    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-modprobe-d\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a052240-fd7b-4102-9bb4-b2b9506f9961-host-slash\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-system-cni-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a052240-fd7b-4102-9bb4-b2b9506f9961-iptables-alerter-script\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.181589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-cnibin\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-hosts-file\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f4bbc94-91c8-4a0d-bc00-ea9be56778d2-agent-certs\") pod \"konnectivity-agent-jnrr6\" (UID: \"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2\") " pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-device-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.181302    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.181328    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.181372    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret podName:636333a8-2370-4e1f-acc9-4ad49e62fcfc nodeName:}" failed. No retries permitted until 2026-07-13 20:19:17.681349363 +0000 UTC m=+3.033958211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret") pod "global-pull-secret-syncer-hqxqb" (UID: "636333a8-2370-4e1f-acc9-4ad49e62fcfc") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28f13592-025b-493b-a761-aab0e1bfda3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181548    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-device-dir\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.181733    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs podName:1347ce06-e218-4be5-9dd2-5db94f75ff35 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:17.681715608 +0000 UTC m=+3.034324435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs") pod "network-metrics-daemon-nrn7d" (UID: "1347ce06-e218-4be5-9dd2-5db94f75ff35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-kubernetes\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181784    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/636333a8-2370-4e1f-acc9-4ad49e62fcfc-dbus\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.182464 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-run\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-conf-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-multus-conf-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b9111d6-4502-4d2d-8594-89f4281983f3-tmp\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b192fbb-a258-4033-881f-1365ace9d445-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.181812    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-kubernetes\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.182051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b9111d6-4502-4d2d-8594-89f4281983f3-run\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.182085    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/636333a8-2370-4e1f-acc9-4ad49e62fcfc-dbus\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.183320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.182163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a052240-fd7b-4102-9bb4-b2b9506f9961-iptables-alerter-script\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.183881 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.183550    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28f13592-025b-493b-a761-aab0e1bfda3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.184614 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.184584    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b9111d6-4502-4d2d-8594-89f4281983f3-etc-tuned\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.185794 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.185767    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf87b37-94c8-4788-b01b-bd97a1ec5c63-system-cni-dir\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.185898 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.185796    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f4bbc94-91c8-4a0d-bc00-ea9be56778d2-agent-certs\") pod \"konnectivity-agent-jnrr6\" (UID: \"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2\") " pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:17.191201 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.191180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngdk\" (UniqueName: \"kubernetes.io/projected/7b192fbb-a258-4033-881f-1365ace9d445-kube-api-access-wngdk\") pod \"aws-ebs-csi-driver-node-p7942\" (UID: \"7b192fbb-a258-4033-881f-1365ace9d445\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.192258 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.192216    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqw6\" (UniqueName: \"kubernetes.io/projected/baf87b37-94c8-4788-b01b-bd97a1ec5c63-kube-api-access-wjqw6\") pod \"multus-v25wz\" (UID: \"baf87b37-94c8-4788-b01b-bd97a1ec5c63\") " pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.192891 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.192716    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56r6n\" (UniqueName: \"kubernetes.io/projected/7a052240-fd7b-4102-9bb4-b2b9506f9961-kube-api-access-56r6n\") pod \"iptables-alerter-9tj22\" (UID: \"7a052240-fd7b-4102-9bb4-b2b9506f9961\") " pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.193103 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.193082    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:17.193188 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.193113    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:17.193188 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.193129    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qvn9k for pod openshift-network-diagnostics/network-check-target-r557g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:17.193295 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.193199    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k podName:9556dff5-b188-4ff9-aaf2-dc315a1ba735 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:17.693182472 +0000 UTC m=+3.045791312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qvn9k" (UniqueName: "kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k") pod "network-check-target-r557g" (UID: "9556dff5-b188-4ff9-aaf2-dc315a1ba735") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:17.194248 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.194230    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bgc\" (UniqueName: \"kubernetes.io/projected/6b9111d6-4502-4d2d-8594-89f4281983f3-kube-api-access-75bgc\") pod \"tuned-lxj57\" (UID: \"6b9111d6-4502-4d2d-8594-89f4281983f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.195850 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.195536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sf76\" (UniqueName: \"kubernetes.io/projected/1347ce06-e218-4be5-9dd2-5db94f75ff35-kube-api-access-7sf76\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:17.195850 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.195824    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e-kube-api-access-8hlvb\") pod \"node-resolver-gzfvp\" (UID: \"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e\") " pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.197136 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.197111    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5gj\" (UniqueName: \"kubernetes.io/projected/28f13592-025b-493b-a761-aab0e1bfda3c-kube-api-access-7z5gj\") pod \"multus-additional-cni-plugins-hc99x\" (UID: \"28f13592-025b-493b-a761-aab0e1bfda3c\") " pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.289800 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.289715    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:17.300092 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.300063    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-68pm9"
Jul 13 20:19:17.309220 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.309196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v25wz"
Jul 13 20:19:17.318109 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.318090    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9tj22"
Jul 13 20:19:17.328655 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.328633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:17.336563 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.336534    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hc99x"
Jul 13 20:19:17.345499 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.345477    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942"
Jul 13 20:19:17.353361 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.353343    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lxj57"
Jul 13 20:19:17.360161 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.360136    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzfvp"
Jul 13 20:19:17.686747 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.686633    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:17.686747 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.686710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:17.686971 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.686805    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:17.686971 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.686899    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs podName:1347ce06-e218-4be5-9dd2-5db94f75ff35 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:18.686874272 +0000 UTC m=+4.039483110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs") pod "network-metrics-daemon-nrn7d" (UID: "1347ce06-e218-4be5-9dd2-5db94f75ff35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:17.686971 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.686907    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:17.687175 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.686976    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret podName:636333a8-2370-4e1f-acc9-4ad49e62fcfc nodeName:}" failed. No retries permitted until 2026-07-13 20:19:18.686955794 +0000 UTC m=+4.039564633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret") pod "global-pull-secret-syncer-hqxqb" (UID: "636333a8-2370-4e1f-acc9-4ad49e62fcfc") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:17.787863 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:17.787824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:17.788100 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.788072    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:17.788169 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.788111    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:17.788169 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.788126    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qvn9k for pod openshift-network-diagnostics/network-check-target-r557g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:17.788261 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:17.788188    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k podName:9556dff5-b188-4ff9-aaf2-dc315a1ba735 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:18.788169034 +0000 UTC m=+4.140777871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvn9k" (UniqueName: "kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k") pod "network-check-target-r557g" (UID: "9556dff5-b188-4ff9-aaf2-dc315a1ba735") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:18.051210 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.051138    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:14:17 +0000 UTC" deadline="2028-02-16 22:49:53.635191406 +0000 UTC"
Jul 13 20:19:18.051210 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.051205    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13994h30m35.583991503s"
Jul 13 20:19:18.125116 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.125082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:18.125269 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.125225    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:18.210892 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.210667    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f13592_025b_493b_a761_aab0e1bfda3c.slice/crio-4fe4f4f0b83c38365cf313359e67ff29aed0e8b08b9626da1e8ac6c8d0c99a26 WatchSource:0}: Error finding container 4fe4f4f0b83c38365cf313359e67ff29aed0e8b08b9626da1e8ac6c8d0c99a26: Status 404 returned error can't find the container with id 4fe4f4f0b83c38365cf313359e67ff29aed0e8b08b9626da1e8ac6c8d0c99a26
Jul 13 20:19:18.211805 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.211777    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df6f3a4_09b8_49bf_b87c_b8bc60e31c6e.slice/crio-6602995342b9e549af0df0e73781772c136ac883c9e59042d6b87083aea59bc8 WatchSource:0}: Error finding container 6602995342b9e549af0df0e73781772c136ac883c9e59042d6b87083aea59bc8: Status 404 returned error can't find the container with id 6602995342b9e549af0df0e73781772c136ac883c9e59042d6b87083aea59bc8
Jul 13 20:19:18.212760 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.212736    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4bbc94_91c8_4a0d_bc00_ea9be56778d2.slice/crio-c107c33ad223706a70bbaefbeeaca039dc8414be2fd7213d481efde0b7c71a82 WatchSource:0}: Error finding container c107c33ad223706a70bbaefbeeaca039dc8414be2fd7213d481efde0b7c71a82: Status 404 returned error can't find the container with id c107c33ad223706a70bbaefbeeaca039dc8414be2fd7213d481efde0b7c71a82
Jul 13 20:19:18.215565 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.215536    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6498e489_3b9d_4ca6_8191_b2cd12b9c37a.slice/crio-54e0e39829fb22477a59c90c30a174d2629511a8f92fb55d2d105bdcbe5701bd WatchSource:0}: Error finding container 54e0e39829fb22477a59c90c30a174d2629511a8f92fb55d2d105bdcbe5701bd: Status 404 returned error can't find the container with id 54e0e39829fb22477a59c90c30a174d2629511a8f92fb55d2d105bdcbe5701bd
Jul 13 20:19:18.215757 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.215722    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b192fbb_a258_4033_881f_1365ace9d445.slice/crio-401a4a8504680a569f06669cbec0ad0c924d4305d307f8d36dff1f6381c619b8 WatchSource:0}: Error finding container 401a4a8504680a569f06669cbec0ad0c924d4305d307f8d36dff1f6381c619b8: Status 404 returned error can't find the container with id 401a4a8504680a569f06669cbec0ad0c924d4305d307f8d36dff1f6381c619b8
Jul 13 20:19:18.218375 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.218347    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a052240_fd7b_4102_9bb4_b2b9506f9961.slice/crio-24e47cb459da7247294dbb24f0a3d5372d9b37ac06b69b9ede60f1382eb71122 WatchSource:0}: Error finding container 24e47cb459da7247294dbb24f0a3d5372d9b37ac06b69b9ede60f1382eb71122: Status 404 returned error can't find the container with id 24e47cb459da7247294dbb24f0a3d5372d9b37ac06b69b9ede60f1382eb71122
Jul 13 20:19:18.240859 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.240837    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9111d6_4502_4d2d_8594_89f4281983f3.slice/crio-a008c956818a949a4514c61090fc9ffd83bc801cbe09fc76eb25718fad061ef9 WatchSource:0}: Error finding container a008c956818a949a4514c61090fc9ffd83bc801cbe09fc76eb25718fad061ef9: Status 404 returned error can't find the container with id a008c956818a949a4514c61090fc9ffd83bc801cbe09fc76eb25718fad061ef9
Jul 13 20:19:18.242346 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.242293    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf87b37_94c8_4788_b01b_bd97a1ec5c63.slice/crio-28e835d5587aa50e819335e57941b42bd2a6f4eee760ff262b23ad6406496653 WatchSource:0}: Error finding container 28e835d5587aa50e819335e57941b42bd2a6f4eee760ff262b23ad6406496653: Status 404 returned error can't find the container with id 28e835d5587aa50e819335e57941b42bd2a6f4eee760ff262b23ad6406496653
Jul 13 20:19:18.243406 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:18.243387    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de3016b_df9e_4924_8f6b_5f42bbc831d8.slice/crio-e49dedf27fb3d3fb6df62f2092eca017bd3253479fcf3cc6d2258491ab5ddb91 WatchSource:0}: Error finding container e49dedf27fb3d3fb6df62f2092eca017bd3253479fcf3cc6d2258491ab5ddb91: Status 404 returned error can't find the container with id e49dedf27fb3d3fb6df62f2092eca017bd3253479fcf3cc6d2258491ab5ddb91
Jul 13 20:19:18.621980 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.621893    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:18.697393 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.697184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:18.697393 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.697258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:18.697393 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.697375    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:18.697701 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.697409    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:18.697701 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.697471    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs podName:1347ce06-e218-4be5-9dd2-5db94f75ff35 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:20.697450147 +0000 UTC m=+6.050058985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs") pod "network-metrics-daemon-nrn7d" (UID: "1347ce06-e218-4be5-9dd2-5db94f75ff35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:18.697701 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.697493    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret podName:636333a8-2370-4e1f-acc9-4ad49e62fcfc nodeName:}" failed. No retries permitted until 2026-07-13 20:19:20.697482649 +0000 UTC m=+6.050091471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret") pod "global-pull-secret-syncer-hqxqb" (UID: "636333a8-2370-4e1f-acc9-4ad49e62fcfc") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:18.798585 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.797910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:18.798585 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.798122    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:18.798585 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.798147    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:18.798585 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.798161    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qvn9k for pod openshift-network-diagnostics/network-check-target-r557g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:18.798585 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:18.798232    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k podName:9556dff5-b188-4ff9-aaf2-dc315a1ba735 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:20.798207042 +0000 UTC m=+6.150815881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvn9k" (UniqueName: "kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k") pod "network-check-target-r557g" (UID: "9556dff5-b188-4ff9-aaf2-dc315a1ba735") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:18.839390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:18.839357    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:19.051735 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.051667    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:14:17 +0000 UTC" deadline="2028-03-03 09:28:25.344196254 +0000 UTC"
Jul 13 20:19:19.051735 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.051706    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14365h9m6.292495277s"
Jul 13 20:19:19.125776 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.125229    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:19.125776 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:19.125368    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:19.126371 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.126202    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:19.126371 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:19.126332    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:19.145459 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.145389    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnrr6" event={"ID":"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2","Type":"ContainerStarted","Data":"c107c33ad223706a70bbaefbeeaca039dc8414be2fd7213d481efde0b7c71a82"}
Jul 13 20:19:19.151366 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.151142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerStarted","Data":"4fe4f4f0b83c38365cf313359e67ff29aed0e8b08b9626da1e8ac6c8d0c99a26"}
Jul 13 20:19:19.171038 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.170976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal" event={"ID":"e3a758c83d40d2b976cf492c86764c97","Type":"ContainerStarted","Data":"a375f4c21d8af16123f738cab04a022e3d492f87cec4a354904f67a0a4b4cd4b"}
Jul 13 20:19:19.200185 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.200152    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v25wz" event={"ID":"baf87b37-94c8-4788-b01b-bd97a1ec5c63","Type":"ContainerStarted","Data":"28e835d5587aa50e819335e57941b42bd2a6f4eee760ff262b23ad6406496653"}
Jul 13 20:19:19.208162 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.208131    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9tj22" event={"ID":"7a052240-fd7b-4102-9bb4-b2b9506f9961","Type":"ContainerStarted","Data":"24e47cb459da7247294dbb24f0a3d5372d9b37ac06b69b9ede60f1382eb71122"}
Jul 13 20:19:19.212812 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.212784    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942" event={"ID":"7b192fbb-a258-4033-881f-1365ace9d445","Type":"ContainerStarted","Data":"401a4a8504680a569f06669cbec0ad0c924d4305d307f8d36dff1f6381c619b8"}
Jul 13 20:19:19.219398 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.219269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzfvp" event={"ID":"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e","Type":"ContainerStarted","Data":"6602995342b9e549af0df0e73781772c136ac883c9e59042d6b87083aea59bc8"}
Jul 13 20:19:19.235055 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.234269    2575 generic.go:356] "Generic (PLEG): container finished" podID="696cfbde432fcad9786b56905bc126bd" containerID="a9e74174c654496e716f7fc8ffaad8fab40d2b1cecbfc1347f9e91f20eef65e8" exitCode=0
Jul 13 20:19:19.235055 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.234342    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal" event={"ID":"696cfbde432fcad9786b56905bc126bd","Type":"ContainerDied","Data":"a9e74174c654496e716f7fc8ffaad8fab40d2b1cecbfc1347f9e91f20eef65e8"}
Jul 13 20:19:19.252436 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.252364    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-233.ec2.internal" podStartSLOduration=3.252346554 podStartE2EDuration="3.252346554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:19.188487899 +0000 UTC m=+4.541096749" watchObservedRunningTime="2026-07-13 20:19:19.252346554 +0000 UTC m=+4.604955399"
Jul 13 20:19:19.256482 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.256456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"e49dedf27fb3d3fb6df62f2092eca017bd3253479fcf3cc6d2258491ab5ddb91"}
Jul 13 20:19:19.263665 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.263590    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lxj57" event={"ID":"6b9111d6-4502-4d2d-8594-89f4281983f3","Type":"ContainerStarted","Data":"a008c956818a949a4514c61090fc9ffd83bc801cbe09fc76eb25718fad061ef9"}
Jul 13 20:19:19.275864 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:19.275834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-68pm9" event={"ID":"6498e489-3b9d-4ca6-8191-b2cd12b9c37a","Type":"ContainerStarted","Data":"54e0e39829fb22477a59c90c30a174d2629511a8f92fb55d2d105bdcbe5701bd"}
Jul 13 20:19:20.125466 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:20.124827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:20.125466 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.124993    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:20.295974 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:20.295881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal" event={"ID":"696cfbde432fcad9786b56905bc126bd","Type":"ContainerStarted","Data":"67acc925a8657d00b42bed3f664a3b31a754dcf99450d382ecd831048340fe99"}
Jul 13 20:19:20.716332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:20.716290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:20.716520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:20.716350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:20.716520 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.716515    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:20.716656 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.716583    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret podName:636333a8-2370-4e1f-acc9-4ad49e62fcfc nodeName:}" failed. No retries permitted until 2026-07-13 20:19:24.716563161 +0000 UTC m=+10.069171986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret") pod "global-pull-secret-syncer-hqxqb" (UID: "636333a8-2370-4e1f-acc9-4ad49e62fcfc") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:20.716656 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.716651    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:20.716792 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.716684    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs podName:1347ce06-e218-4be5-9dd2-5db94f75ff35 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:24.716671067 +0000 UTC m=+10.069279912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs") pod "network-metrics-daemon-nrn7d" (UID: "1347ce06-e218-4be5-9dd2-5db94f75ff35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:20.817086 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:20.817030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:20.817287 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.817211    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:20.817287 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.817232    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:20.817287 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.817245    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qvn9k for pod openshift-network-diagnostics/network-check-target-r557g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:20.817473 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:20.817308    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k podName:9556dff5-b188-4ff9-aaf2-dc315a1ba735 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:24.817288359 +0000 UTC m=+10.169897197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvn9k" (UniqueName: "kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k") pod "network-check-target-r557g" (UID: "9556dff5-b188-4ff9-aaf2-dc315a1ba735") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:21.126088 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:21.125206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:21.126088 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:21.125344    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:21.126088 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:21.125764    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:21.126088 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:21.125878    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:22.124295 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:22.124248    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:22.124496 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:22.124410    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:23.127112 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:23.127073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:23.127520 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:23.127202    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:23.127520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:23.127265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:23.127520 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:23.127363    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:24.124258 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:24.124217    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:24.124421 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.124375    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:24.751805 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:24.751767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:24.752213 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:24.751876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:24.752213 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.752006    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:24.752213 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.752071    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs podName:1347ce06-e218-4be5-9dd2-5db94f75ff35 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:32.752051119 +0000 UTC m=+18.104659942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs") pod "network-metrics-daemon-nrn7d" (UID: "1347ce06-e218-4be5-9dd2-5db94f75ff35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:24.752476 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.752459    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:24.752553 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.752511    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret podName:636333a8-2370-4e1f-acc9-4ad49e62fcfc nodeName:}" failed. No retries permitted until 2026-07-13 20:19:32.752496866 +0000 UTC m=+18.105105687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret") pod "global-pull-secret-syncer-hqxqb" (UID: "636333a8-2370-4e1f-acc9-4ad49e62fcfc") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:24.852611 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:24.852569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:24.852778 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.852758    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:24.852828 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.852786    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:24.852828 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.852796    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qvn9k for pod openshift-network-diagnostics/network-check-target-r557g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:24.852917 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:24.852853    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k podName:9556dff5-b188-4ff9-aaf2-dc315a1ba735 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:32.852833222 +0000 UTC m=+18.205442075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvn9k" (UniqueName: "kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k") pod "network-check-target-r557g" (UID: "9556dff5-b188-4ff9-aaf2-dc315a1ba735") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:25.126378 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:25.125855    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:25.126378 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:25.125938    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:25.126378 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:25.125963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:25.126378 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:25.126062    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:26.124148 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:26.124113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:26.124586 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:26.124245    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:27.126817 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:27.126173    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:27.126817 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:27.126294    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:27.126817 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:27.126648    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:27.126817 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:27.126737    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:28.124298 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:28.124266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:28.124488 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:28.124393    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:29.124981 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:29.124950    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:29.125421 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:29.124950    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:29.125421 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:29.125089    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:29.125421 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:29.125155    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:30.124895 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:30.124861    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:30.125069 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:30.124973    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:31.124251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:31.124213    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:31.124703 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:31.124210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:31.124703 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:31.124377    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:31.124703 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:31.124421    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:32.124176 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:32.124142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:32.124408 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.124264    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:32.813528 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:32.813494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:32.813703 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:32.813543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:32.813703 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.813666    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:32.813703 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.813667    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:32.813835 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.813717    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret podName:636333a8-2370-4e1f-acc9-4ad49e62fcfc nodeName:}" failed. No retries permitted until 2026-07-13 20:19:48.813703824 +0000 UTC m=+34.166312645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret") pod "global-pull-secret-syncer-hqxqb" (UID: "636333a8-2370-4e1f-acc9-4ad49e62fcfc") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:32.813835 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.813730    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs podName:1347ce06-e218-4be5-9dd2-5db94f75ff35 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:48.813723354 +0000 UTC m=+34.166332174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs") pod "network-metrics-daemon-nrn7d" (UID: "1347ce06-e218-4be5-9dd2-5db94f75ff35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:32.913971 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:32.913928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:32.914141 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.914109    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:32.914141 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.914131    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:32.914243 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.914145    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qvn9k for pod openshift-network-diagnostics/network-check-target-r557g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:32.914243 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:32.914191    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k podName:9556dff5-b188-4ff9-aaf2-dc315a1ba735 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:48.914178985 +0000 UTC m=+34.266787819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvn9k" (UniqueName: "kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k") pod "network-check-target-r557g" (UID: "9556dff5-b188-4ff9-aaf2-dc315a1ba735") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:33.125275 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:33.124722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:33.125275 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:33.124860    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:33.125275 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:33.124919    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:33.125275 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:33.125066    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:34.124538 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:34.124499    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:34.124736 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:34.124663    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:35.126018 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:35.125014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:35.126018 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:35.125145    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:35.126018 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:35.125207    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:35.126018 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:35.125327    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:36.124553 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:36.124517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:36.124714 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:36.124634    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:37.124283 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:37.124245    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:37.124283 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:37.124282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:37.124961 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:37.124358    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:37.124961 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:37.124508    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:38.125022 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.124684    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:38.125637 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:38.125123    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:38.342592 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.342556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lxj57" event={"ID":"6b9111d6-4502-4d2d-8594-89f4281983f3","Type":"ContainerStarted","Data":"3de31e64b2a14b52e88332be687c2c5979837cc8ee00bbc9bf43dad9d7b9690a"}
Jul 13 20:19:38.344575 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.344236    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-68pm9" event={"ID":"6498e489-3b9d-4ca6-8191-b2cd12b9c37a","Type":"ContainerStarted","Data":"5af0f4080923a8b91963ccb17ad4beec36fedcbea63344ebc5545356148cff78"}
Jul 13 20:19:38.345791 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.345749    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnrr6" event={"ID":"3f4bbc94-91c8-4a0d-bc00-ea9be56778d2","Type":"ContainerStarted","Data":"4b4b5d9fdd633cfdfa6694bbd218eb9fab52bfdf433182bf15e4574aff34fbdf"}
Jul 13 20:19:38.347411 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.347384    2575 generic.go:356] "Generic (PLEG): container finished" podID="28f13592-025b-493b-a761-aab0e1bfda3c" containerID="b0285e3d9ce4f87f51279ff0ae167c6e4f57767882e5fa4476bdccd86c531763" exitCode=0
Jul 13 20:19:38.347522 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.347474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerDied","Data":"b0285e3d9ce4f87f51279ff0ae167c6e4f57767882e5fa4476bdccd86c531763"}
Jul 13 20:19:38.348943 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.348917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v25wz" event={"ID":"baf87b37-94c8-4788-b01b-bd97a1ec5c63","Type":"ContainerStarted","Data":"fd3948c15b3f2be101f831a5a949c8f26413886b54a03b90533e70413c0f0d90"}
Jul 13 20:19:38.350720 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.350700    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942" event={"ID":"7b192fbb-a258-4033-881f-1365ace9d445","Type":"ContainerStarted","Data":"0fcc128efa8b8a10c8966d34de1c17f20b72d2958e1016ea843006eaa183ee2c"}
Jul 13 20:19:38.355249 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.355226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzfvp" event={"ID":"5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e","Type":"ContainerStarted","Data":"94efb9e1735627d8a54aff56e47e7f4c400836323134330ccb05795de4975f25"}
Jul 13 20:19:38.360930 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.360909    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:19:38.361319 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.361301    2575 generic.go:356] "Generic (PLEG): container finished" podID="1de3016b-df9e-4924-8f6b-5f42bbc831d8" containerID="433e7c05a8b50ed15923f064e6e33b14bb5b0be510c77aa9f38668657417532c" exitCode=1
Jul 13 20:19:38.361387 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.361331    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"9c16ee5894dbff8c6f2885221207c7c26707c6d62060611f7cf70ccc1ea8c954"}
Jul 13 20:19:38.361387 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.361379    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"ac57253fc51dc08ca155bdeebcb3a76af4d5297d7ce183f54c260a8c0b4157d8"}
Jul 13 20:19:38.361475 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.361422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"550aa55eab88d67dee07696947c077dec2f0bd359d2b20b01c9bee323936287b"}
Jul 13 20:19:38.361475 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.361450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"ac3d0dde0c5183ffc077198ceb89ffcb17c2ec22f65f319d3ee640c665e1cdaf"}
Jul 13 20:19:38.361544 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.361475    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerDied","Data":"433e7c05a8b50ed15923f064e6e33b14bb5b0be510c77aa9f38668657417532c"}
Jul 13 20:19:38.361544 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.361502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"dca00fb553fe48d223704a7d3fda4edde78756a8ec232c794752ce8dc9f0b583"}
Jul 13 20:19:38.370029 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.369965    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lxj57" podStartSLOduration=4.144918889 podStartE2EDuration="23.369954279s" totalImagesPullingTime="19.22503539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:38.369527343 +0000 UTC m=+23.722136187" watchObservedRunningTime="2026-07-13 20:19:38.369954279 +0000 UTC m=+23.722563121"
Jul 13 20:19:38.370182 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.370150    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-233.ec2.internal" podStartSLOduration=22.37014198 podStartE2EDuration="22.37014198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:20.31801527 +0000 UTC m=+5.670624118" watchObservedRunningTime="2026-07-13 20:19:38.37014198 +0000 UTC m=+23.722750826"
Jul 13 20:19:38.388221 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.388179    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gzfvp" podStartSLOduration=4.147739039 podStartE2EDuration="23.388162436s" totalImagesPullingTime="19.240423397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:38.387866492 +0000 UTC m=+23.740475367" watchObservedRunningTime="2026-07-13 20:19:38.388162436 +0000 UTC m=+23.740771282"
Jul 13 20:19:38.405639 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.405595    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-68pm9" podStartSLOduration=4.168950362 podStartE2EDuration="23.405583618s" totalImagesPullingTime="19.236633256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:38.405143315 +0000 UTC m=+23.757752196" watchObservedRunningTime="2026-07-13 20:19:38.405583618 +0000 UTC m=+23.758192461"
Jul 13 20:19:38.433766 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.433717    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v25wz" podStartSLOduration=4.164651664 podStartE2EDuration="23.433703293s" totalImagesPullingTime="19.269051629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:38.433362387 +0000 UTC m=+23.785971426" watchObservedRunningTime="2026-07-13 20:19:38.433703293 +0000 UTC m=+23.786312136"
Jul 13 20:19:38.475520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.475474    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jnrr6" podStartSLOduration=12.370995485 podStartE2EDuration="23.475456093s" totalImagesPullingTime="11.104460608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:38.474840617 +0000 UTC m=+23.827449460" watchObservedRunningTime="2026-07-13 20:19:38.475456093 +0000 UTC m=+23.828064937"
Jul 13 20:19:38.860052 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:38.859946    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 20:19:39.040777 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:39.040618    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T20:19:38.859971799Z","UUID":"9916c83f-9273-4e45-b828-931f97568198","Handler":null,"Name":"","Endpoint":""}
Jul 13 20:19:39.042233 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:39.042212    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 20:19:39.042351 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:39.042247    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 20:19:39.124417 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:39.124387    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:39.124649 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:39.124395    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:39.124649 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:39.124514    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:39.124649 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:39.124611    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:39.366341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:39.366260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9tj22" event={"ID":"7a052240-fd7b-4102-9bb4-b2b9506f9961","Type":"ContainerStarted","Data":"942708617bf353baca64e6b93edd9b8608759405bae49c8e034cca1523880a40"}
Jul 13 20:19:39.369058 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:39.369026    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942" event={"ID":"7b192fbb-a258-4033-881f-1365ace9d445","Type":"ContainerStarted","Data":"2b3908a192a6028b35833bc4904b4a19abcfcf385ebc9b74f7dffede26227cc0"}
Jul 13 20:19:40.124331 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:40.124063    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:40.124490 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:40.124395    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:40.372839 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:40.372803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942" event={"ID":"7b192fbb-a258-4033-881f-1365ace9d445","Type":"ContainerStarted","Data":"108cc068fc00cfbb5c4f62b3c9565abd96105df4a89e233ad375592b1358deb6"}
Jul 13 20:19:40.407456 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:40.407403    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9tj22" podStartSLOduration=6.193067339 podStartE2EDuration="25.407384552s" totalImagesPullingTime="19.214317213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:39.385327833 +0000 UTC m=+24.737936676" watchObservedRunningTime="2026-07-13 20:19:40.407384552 +0000 UTC m=+25.759993396"
Jul 13 20:19:40.407708 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:40.407683    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7942" podStartSLOduration=4.120116245 podStartE2EDuration="25.407675234s" totalImagesPullingTime="21.287558989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:40.407068354 +0000 UTC m=+25.759677198" watchObservedRunningTime="2026-07-13 20:19:40.407675234 +0000 UTC m=+25.760284081"
Jul 13 20:19:41.124695 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:41.124659    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:41.124898 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:41.124664    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:41.124898 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:41.124798    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:41.124898 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:41.124885    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:41.378541 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:41.378467    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:19:41.379190 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:41.378913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"e2f06e7c63387fe4c5af8708a723e62356749da2cdc969b03e4fb0aa6634979c"}
Jul 13 20:19:42.124713 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:42.124675    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:42.124885 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:42.124790    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:42.966507 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:42.966345    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:42.967127 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:42.967003    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:43.124345 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.124309    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:43.124538 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:43.124424    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:43.124607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.124533    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:43.124689 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:43.124669    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:43.385777 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.385708    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:19:43.386125 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.386098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"c62f0516a20b0ca068651764ea8b31899410c88f1ddb3832514fe3f139e2b84f"}
Jul 13 20:19:43.386562 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.386536    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:43.386711 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.386567    2575 scope.go:117] "RemoveContainer" containerID="433e7c05a8b50ed15923f064e6e33b14bb5b0be510c77aa9f38668657417532c"
Jul 13 20:19:43.386711 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.386647    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:43.388031 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.388002    2575 generic.go:356] "Generic (PLEG): container finished" podID="28f13592-025b-493b-a761-aab0e1bfda3c" containerID="4f735ad9a67d7e2021e85ffb990ea72c93bda03c296715ff4e0a14ae51e2f576" exitCode=0
Jul 13 20:19:43.388156 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.388087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerDied","Data":"4f735ad9a67d7e2021e85ffb990ea72c93bda03c296715ff4e0a14ae51e2f576"}
Jul 13 20:19:43.404734 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.404714    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:43.404862 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.404851    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:43.824163 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.824128    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:43.825009 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.824965    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jnrr6"
Jul 13 20:19:43.873777 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:43.873738    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:19:44.124534 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.124463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:44.124918 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:44.124589    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:44.396341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.396317    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:19:44.396770 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.396732    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" event={"ID":"1de3016b-df9e-4924-8f6b-5f42bbc831d8","Type":"ContainerStarted","Data":"10958034c36dfe398e522b6775ec43a44d17aac0c9b0ea44506320516bc83e3c"}
Jul 13 20:19:44.434153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.433959    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74" podStartSLOduration=10.158859574 podStartE2EDuration="29.433943031s" totalImagesPullingTime="19.275083457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:44.433361927 +0000 UTC m=+29.785970788" watchObservedRunningTime="2026-07-13 20:19:44.433943031 +0000 UTC m=+29.786551878"
Jul 13 20:19:44.617862 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.617833    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hqxqb"]
Jul 13 20:19:44.618056 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.618006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:44.618143 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:44.618121    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:44.622429 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.622406    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r557g"]
Jul 13 20:19:44.622534 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.622506    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:44.622614 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:44.622595    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:44.624699 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.624680    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nrn7d"]
Jul 13 20:19:44.624796 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:44.624783    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:44.624937 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:44.624919    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:45.400677 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:45.400638    2575 generic.go:356] "Generic (PLEG): container finished" podID="28f13592-025b-493b-a761-aab0e1bfda3c" containerID="f0085acc3cbef88b6bb8eb8b16a1164c9a286be173e6de275c132787721d1084" exitCode=0
Jul 13 20:19:45.401204 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:45.400692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerDied","Data":"f0085acc3cbef88b6bb8eb8b16a1164c9a286be173e6de275c132787721d1084"}
Jul 13 20:19:46.124261 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:46.124223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:46.124261 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:46.124253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:46.124496 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:46.124263    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:46.124496 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:46.124331    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:46.124496 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:46.124430    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:46.124633 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:46.124518    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:47.407770 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:47.407738    2575 generic.go:356] "Generic (PLEG): container finished" podID="28f13592-025b-493b-a761-aab0e1bfda3c" containerID="0b29e7611acdb32c71166cbaef1515810db2a758067b19969e4f78a206f29c5d" exitCode=0
Jul 13 20:19:47.408304 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:47.407796    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerDied","Data":"0b29e7611acdb32c71166cbaef1515810db2a758067b19969e4f78a206f29c5d"}
Jul 13 20:19:48.124047 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:48.124017    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:48.124228 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:48.124045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:48.124228 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.124139    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:48.124228 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.124198    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:48.124390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:48.124246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:48.124390 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.124294    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:48.829748 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:48.829710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:48.830231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:48.829755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:48.830231 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.829892    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:48.830231 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.829896    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:48.830231 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.829963    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret podName:636333a8-2370-4e1f-acc9-4ad49e62fcfc nodeName:}" failed. No retries permitted until 2026-07-13 20:20:20.82994654 +0000 UTC m=+66.182555379 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret") pod "global-pull-secret-syncer-hqxqb" (UID: "636333a8-2370-4e1f-acc9-4ad49e62fcfc") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:48.830231 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.829984    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs podName:1347ce06-e218-4be5-9dd2-5db94f75ff35 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:20.829974141 +0000 UTC m=+66.182582969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs") pod "network-metrics-daemon-nrn7d" (UID: "1347ce06-e218-4be5-9dd2-5db94f75ff35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:48.931148 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:48.931108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:48.931331 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.931298    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:48.931331 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.931322    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:48.931413 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.931335    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qvn9k for pod openshift-network-diagnostics/network-check-target-r557g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:48.931413 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:48.931394    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k podName:9556dff5-b188-4ff9-aaf2-dc315a1ba735 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:20.931375943 +0000 UTC m=+66.283984777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvn9k" (UniqueName: "kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k") pod "network-check-target-r557g" (UID: "9556dff5-b188-4ff9-aaf2-dc315a1ba735") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:50.124797 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:50.124605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:50.125231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:50.124636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:50.125231 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:50.124884    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqxqb" podUID="636333a8-2370-4e1f-acc9-4ad49e62fcfc"
Jul 13 20:19:50.125231 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:50.124972    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r557g" podUID="9556dff5-b188-4ff9-aaf2-dc315a1ba735"
Jul 13 20:19:50.125231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:50.124657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:50.125231 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:50.125074    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nrn7d" podUID="1347ce06-e218-4be5-9dd2-5db94f75ff35"
Jul 13 20:19:51.514693 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.514672    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-233.ec2.internal" event="NodeReady"
Jul 13 20:19:51.515094 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.514821    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 20:19:51.567739 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.567711    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lrsdq"]
Jul 13 20:19:51.570512 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.570493    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hdctk"]
Jul 13 20:19:51.570619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.570603    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.573384 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.573357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 20:19:51.573487 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.573460    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t5qs6\""
Jul 13 20:19:51.573550 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.573493    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 20:19:51.573550 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.573518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:51.576053 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.576037    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 20:19:51.576400 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.576323    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wmtv8\""
Jul 13 20:19:51.576529 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.576515    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 20:19:51.576685 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.576670    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 20:19:51.586596 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.586572    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdctk"]
Jul 13 20:19:51.587118 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.587098    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lrsdq"]
Jul 13 20:19:51.653420 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.653393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7d4\" (UniqueName: \"kubernetes.io/projected/08728e65-2a79-442c-b3f8-7704552630e2-kube-api-access-ns7d4\") pod \"ingress-canary-hdctk\" (UID: \"08728e65-2a79-442c-b3f8-7704552630e2\") " pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:51.653561 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.653426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd0dafe-9870-40d4-b882-724ab09f3395-tmp-dir\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.653561 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.653453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd0dafe-9870-40d4-b882-724ab09f3395-config-volume\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.653561 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.653472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.653715 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.653584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08728e65-2a79-442c-b3f8-7704552630e2-cert\") pod \"ingress-canary-hdctk\" (UID: \"08728e65-2a79-442c-b3f8-7704552630e2\") " pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:51.653715 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.653623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8k7\" (UniqueName: \"kubernetes.io/projected/6dd0dafe-9870-40d4-b882-724ab09f3395-kube-api-access-tm8k7\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.754756 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.754655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7d4\" (UniqueName: \"kubernetes.io/projected/08728e65-2a79-442c-b3f8-7704552630e2-kube-api-access-ns7d4\") pod \"ingress-canary-hdctk\" (UID: \"08728e65-2a79-442c-b3f8-7704552630e2\") " pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:51.754756 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.754703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd0dafe-9870-40d4-b882-724ab09f3395-tmp-dir\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.754756 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.754745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd0dafe-9870-40d4-b882-724ab09f3395-config-volume\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.755051 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.754769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.755051 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:51.754878    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:51.755051 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.754923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08728e65-2a79-442c-b3f8-7704552630e2-cert\") pod \"ingress-canary-hdctk\" (UID: \"08728e65-2a79-442c-b3f8-7704552630e2\") " pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:51.755051 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:51.754952    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls podName:6dd0dafe-9870-40d4-b882-724ab09f3395 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:52.254929035 +0000 UTC m=+37.607537880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls") pod "dns-default-lrsdq" (UID: "6dd0dafe-9870-40d4-b882-724ab09f3395") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:51.755051 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.755010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8k7\" (UniqueName: \"kubernetes.io/projected/6dd0dafe-9870-40d4-b882-724ab09f3395-kube-api-access-tm8k7\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.755247 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.755157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd0dafe-9870-40d4-b882-724ab09f3395-tmp-dir\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.755368 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.755351    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd0dafe-9870-40d4-b882-724ab09f3395-config-volume\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.759238 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.759217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08728e65-2a79-442c-b3f8-7704552630e2-cert\") pod \"ingress-canary-hdctk\" (UID: \"08728e65-2a79-442c-b3f8-7704552630e2\") " pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:51.778813 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.778777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8k7\" (UniqueName: \"kubernetes.io/projected/6dd0dafe-9870-40d4-b882-724ab09f3395-kube-api-access-tm8k7\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:51.778917 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.778868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7d4\" (UniqueName: \"kubernetes.io/projected/08728e65-2a79-442c-b3f8-7704552630e2-kube-api-access-ns7d4\") pod \"ingress-canary-hdctk\" (UID: \"08728e65-2a79-442c-b3f8-7704552630e2\") " pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:51.887160 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:51.887128    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdctk"
Jul 13 20:19:52.018798 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.018715    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdctk"]
Jul 13 20:19:52.022870 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:52.022849    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08728e65_2a79_442c_b3f8_7704552630e2.slice/crio-e348f9b62fe239da2d705d04e6b8ab23cfa027e54e1ea20e5b8f398dd3aafe84 WatchSource:0}: Error finding container e348f9b62fe239da2d705d04e6b8ab23cfa027e54e1ea20e5b8f398dd3aafe84: Status 404 returned error can't find the container with id e348f9b62fe239da2d705d04e6b8ab23cfa027e54e1ea20e5b8f398dd3aafe84
Jul 13 20:19:52.124203 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.124151    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:19:52.124203 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.124200    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:19:52.124443 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.124157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:19:52.128129 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.128108    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 20:19:52.128262 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.128135    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 20:19:52.128262 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.128139    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 20:19:52.128262 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.128156    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 20:19:52.128262 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.128108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6z4f\""
Jul 13 20:19:52.128444 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.128346    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-899ms\""
Jul 13 20:19:52.258880 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.258847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:52.259077 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:52.258953    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:52.259077 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:52.259021    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls podName:6dd0dafe-9870-40d4-b882-724ab09f3395 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:53.259007209 +0000 UTC m=+38.611616030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls") pod "dns-default-lrsdq" (UID: "6dd0dafe-9870-40d4-b882-724ab09f3395") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:52.420384 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.420294    2575 generic.go:356] "Generic (PLEG): container finished" podID="28f13592-025b-493b-a761-aab0e1bfda3c" containerID="d0ddc93cd7480b9440acbab0e09e14acfed54995b2f1ef13b3370795f58048e9" exitCode=0
Jul 13 20:19:52.420504 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.420388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerDied","Data":"d0ddc93cd7480b9440acbab0e09e14acfed54995b2f1ef13b3370795f58048e9"}
Jul 13 20:19:52.421558 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:52.421536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdctk" event={"ID":"08728e65-2a79-442c-b3f8-7704552630e2","Type":"ContainerStarted","Data":"e348f9b62fe239da2d705d04e6b8ab23cfa027e54e1ea20e5b8f398dd3aafe84"}
Jul 13 20:19:53.267091 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:53.266884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:53.267512 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:53.267056    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:53.267512 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:53.267239    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls podName:6dd0dafe-9870-40d4-b882-724ab09f3395 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:55.267217947 +0000 UTC m=+40.619826771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls") pod "dns-default-lrsdq" (UID: "6dd0dafe-9870-40d4-b882-724ab09f3395") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:53.427475 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:53.427440    2575 generic.go:356] "Generic (PLEG): container finished" podID="28f13592-025b-493b-a761-aab0e1bfda3c" containerID="401a753ac69d0a120aa9d791b01dde598ddd0cd079326905d4beaed737673973" exitCode=0
Jul 13 20:19:53.427634 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:53.427484    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerDied","Data":"401a753ac69d0a120aa9d791b01dde598ddd0cd079326905d4beaed737673973"}
Jul 13 20:19:54.434358 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:54.434264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hc99x" event={"ID":"28f13592-025b-493b-a761-aab0e1bfda3c","Type":"ContainerStarted","Data":"0f0622c6a16c83c839720dc446d3d8755ffbf8069c8a0251b750178c1ae64fc3"}
Jul 13 20:19:54.435721 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:54.435696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdctk" event={"ID":"08728e65-2a79-442c-b3f8-7704552630e2","Type":"ContainerStarted","Data":"0e7fa3ec01e504f2f3f03173c2e9b66a927d4f12e89c65cc8c660c30eedf3374"}
Jul 13 20:19:54.460058 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:54.460010    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hc99x" podStartSLOduration=9.890377347 podStartE2EDuration="39.459979651s" totalImagesPullingTime="29.569602304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:54.458111177 +0000 UTC m=+39.810720038" watchObservedRunningTime="2026-07-13 20:19:54.459979651 +0000 UTC m=+39.812588498"
Jul 13 20:19:54.474585 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:54.474542    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hdctk" podStartSLOduration=1.279514483 podStartE2EDuration="3.474526294s" totalImagesPullingTime="2.195011811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:54.474171158 +0000 UTC m=+39.826780004" watchObservedRunningTime="2026-07-13 20:19:54.474526294 +0000 UTC m=+39.827135136"
Jul 13 20:19:55.282447 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:55.282406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:55.282625 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:55.282570    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:55.282671 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:55.282636    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls podName:6dd0dafe-9870-40d4-b882-724ab09f3395 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:59.282620561 +0000 UTC m=+44.635229385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls") pod "dns-default-lrsdq" (UID: "6dd0dafe-9870-40d4-b882-724ab09f3395") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:58.154931 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.154899    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"]
Jul 13 20:19:58.180748 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.180724    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"
Jul 13 20:19:58.186212 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.186189    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 13 20:19:58.186482 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.186462    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5r4jm\""
Jul 13 20:19:58.186531 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.186479    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 13 20:19:58.190618 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.190595    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"]
Jul 13 20:19:58.304809 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.304772    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65prk\" (UniqueName: \"kubernetes.io/projected/83ba3960-58a8-4464-bbcf-6dd4e9d95911-kube-api-access-65prk\") pod \"migrator-7f59b54f6c-tfhfg\" (UID: \"83ba3960-58a8-4464-bbcf-6dd4e9d95911\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"
Jul 13 20:19:58.405749 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.405664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65prk\" (UniqueName: \"kubernetes.io/projected/83ba3960-58a8-4464-bbcf-6dd4e9d95911-kube-api-access-65prk\") pod \"migrator-7f59b54f6c-tfhfg\" (UID: \"83ba3960-58a8-4464-bbcf-6dd4e9d95911\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"
Jul 13 20:19:58.416224 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.416199    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65prk\" (UniqueName: \"kubernetes.io/projected/83ba3960-58a8-4464-bbcf-6dd4e9d95911-kube-api-access-65prk\") pod \"migrator-7f59b54f6c-tfhfg\" (UID: \"83ba3960-58a8-4464-bbcf-6dd4e9d95911\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"
Jul 13 20:19:58.490170 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.490138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"
Jul 13 20:19:58.634348 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.634315    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg"]
Jul 13 20:19:58.637380 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:19:58.637352    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ba3960_58a8_4464_bbcf_6dd4e9d95911.slice/crio-0a3106f8c57d7cc423cb44810d1f2f4cd2e1e15bbf8525524c3343836a16f6b7 WatchSource:0}: Error finding container 0a3106f8c57d7cc423cb44810d1f2f4cd2e1e15bbf8525524c3343836a16f6b7: Status 404 returned error can't find the container with id 0a3106f8c57d7cc423cb44810d1f2f4cd2e1e15bbf8525524c3343836a16f6b7
Jul 13 20:19:58.821940 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.821895    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kbtzv"]
Jul 13 20:19:58.858406 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.858371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:58.862805 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.862775    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kbtzv"]
Jul 13 20:19:58.869534 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.869514    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 20:19:58.869659 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.869564    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 20:19:58.870666 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.870650    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 20:19:58.870757 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.870699    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 20:19:58.870757 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:58.870746    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nlmqs\""
Jul 13 20:19:59.010787 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.010746    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.010953 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.010800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.010953 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.010888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-crio-socket\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.011131 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.010954    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvx6d\" (UniqueName: \"kubernetes.io/projected/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-api-access-mvx6d\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.011131 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.011008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-data-volume\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.111881 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.111818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.111881 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.111865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-crio-socket\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.112048 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.111967    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:59.112048 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.112038    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-crio-socket\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.112115 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.112071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvx6d\" (UniqueName: \"kubernetes.io/projected/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-api-access-mvx6d\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.112115 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.112080    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:59.612066296 +0000 UTC m=+44.964675117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:59.112191 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.112122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-data-volume\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.112191 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.112173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.112275 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.112263    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:19:59.112308 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.112304    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:59.612292632 +0000 UTC m=+44.964901453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : configmap "kube-rbac-proxy" not found
Jul 13 20:19:59.112420 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.112401    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-data-volume\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.123174 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.123153    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvx6d\" (UniqueName: \"kubernetes.io/projected/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-api-access-mvx6d\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.314514 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.314474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:19:59.314945 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.314655    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:59.314945 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.314725    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls podName:6dd0dafe-9870-40d4-b882-724ab09f3395 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:07.314706283 +0000 UTC m=+52.667315105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls") pod "dns-default-lrsdq" (UID: "6dd0dafe-9870-40d4-b882-724ab09f3395") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:59.448942 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.448899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg" event={"ID":"83ba3960-58a8-4464-bbcf-6dd4e9d95911","Type":"ContainerStarted","Data":"0a3106f8c57d7cc423cb44810d1f2f4cd2e1e15bbf8525524c3343836a16f6b7"}
Jul 13 20:19:59.616458 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.616428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.616619 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:19:59.616470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:19:59.616619 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.616582    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:59.616619 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.616587    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:19:59.616723 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.616643    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:00.616630785 +0000 UTC m=+45.969239606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:59.616723 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:19:59.616659    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:00.616652409 +0000 UTC m=+45.969261230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : configmap "kube-rbac-proxy" not found
Jul 13 20:20:00.313853 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.313817    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-n6nx9"]
Jul 13 20:20:00.325469 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.325445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.329899 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.329876    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 13 20:20:00.330029 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.329979    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 13 20:20:00.331178 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.331148    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 13 20:20:00.331312 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.331258    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 13 20:20:00.331392 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.331377    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vkfjc\""
Jul 13 20:20:00.332243 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.332216    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-n6nx9"]
Jul 13 20:20:00.423136 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.423062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/71e4daec-773e-4190-bf03-cdcdb7b4d313-signing-cabundle\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.423136 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.423109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/71e4daec-773e-4190-bf03-cdcdb7b4d313-signing-key\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.423325 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.423144    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thllr\" (UniqueName: \"kubernetes.io/projected/71e4daec-773e-4190-bf03-cdcdb7b4d313-kube-api-access-thllr\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.452778 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.452742    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg" event={"ID":"83ba3960-58a8-4464-bbcf-6dd4e9d95911","Type":"ContainerStarted","Data":"943475e07a781c3cb4761e750c56fa10293226297e517f91a8f9017a918a33d6"}
Jul 13 20:20:00.452889 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.452801    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg" event={"ID":"83ba3960-58a8-4464-bbcf-6dd4e9d95911","Type":"ContainerStarted","Data":"6d5a665673fc967e6228d829e321c7a8b0098da14e60ef9d8a424d3175846524"}
Jul 13 20:20:00.476247 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.476196    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tfhfg" podStartSLOduration=1.014625797 podStartE2EDuration="2.476180366s" totalImagesPullingTime="1.461554569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:00.474765412 +0000 UTC m=+45.827374254" watchObservedRunningTime="2026-07-13 20:20:00.476180366 +0000 UTC m=+45.828789243"
Jul 13 20:20:00.524091 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.524059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/71e4daec-773e-4190-bf03-cdcdb7b4d313-signing-cabundle\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.524228 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.524108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/71e4daec-773e-4190-bf03-cdcdb7b4d313-signing-key\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.524228 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.524137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thllr\" (UniqueName: \"kubernetes.io/projected/71e4daec-773e-4190-bf03-cdcdb7b4d313-kube-api-access-thllr\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.524720 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.524700    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/71e4daec-773e-4190-bf03-cdcdb7b4d313-signing-cabundle\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.526671 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.526652    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/71e4daec-773e-4190-bf03-cdcdb7b4d313-signing-key\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.533840 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.533815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thllr\" (UniqueName: \"kubernetes.io/projected/71e4daec-773e-4190-bf03-cdcdb7b4d313-kube-api-access-thllr\") pod \"service-ca-79f68b9f4f-n6nx9\" (UID: \"71e4daec-773e-4190-bf03-cdcdb7b4d313\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.625315 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.625273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:00.625315 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.625320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:00.625569 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:00.625430    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:20:00.625569 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:00.625466    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:00.625569 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:00.625489    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:02.625474182 +0000 UTC m=+47.978083002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : configmap "kube-rbac-proxy" not found
Jul 13 20:20:00.625569 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:00.625523    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:02.625507003 +0000 UTC m=+47.978115838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:00.634875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.634849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9"
Jul 13 20:20:00.779820 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:00.779793    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-n6nx9"]
Jul 13 20:20:00.801047 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:00.801008    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e4daec_773e_4190_bf03_cdcdb7b4d313.slice/crio-cb92dbbfa2ddc2d4c88679d72c191b21c02aebf1ef3bc36dacb5dd392cfddd48 WatchSource:0}: Error finding container cb92dbbfa2ddc2d4c88679d72c191b21c02aebf1ef3bc36dacb5dd392cfddd48: Status 404 returned error can't find the container with id cb92dbbfa2ddc2d4c88679d72c191b21c02aebf1ef3bc36dacb5dd392cfddd48
Jul 13 20:20:01.456726 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:01.456695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9" event={"ID":"71e4daec-773e-4190-bf03-cdcdb7b4d313","Type":"ContainerStarted","Data":"cb92dbbfa2ddc2d4c88679d72c191b21c02aebf1ef3bc36dacb5dd392cfddd48"}
Jul 13 20:20:01.615901 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:01.615872    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzfvp_5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e/dns-node-resolver/0.log"
Jul 13 20:20:02.411706 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:02.411674    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-68pm9_6498e489-3b9d-4ca6-8191-b2cd12b9c37a/node-ca/0.log"
Jul 13 20:20:02.640866 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:02.640810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:02.641288 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:02.640887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:02.641288 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:02.640946    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:20:02.641288 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:02.641024    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:06.641009226 +0000 UTC m=+51.993618065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : configmap "kube-rbac-proxy" not found
Jul 13 20:20:02.641288 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:02.641031    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:02.641288 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:02.641110    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:06.641094182 +0000 UTC m=+51.993703020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:03.614236 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:03.614209    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hdctk_08728e65-2a79-442c-b3f8-7704552630e2/serve-healthcheck-canary/0.log"
Jul 13 20:20:03.816170 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:03.816148    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-tfhfg_83ba3960-58a8-4464-bbcf-6dd4e9d95911/migrator/0.log"
Jul 13 20:20:04.013034 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:04.013007    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-tfhfg_83ba3960-58a8-4464-bbcf-6dd4e9d95911/graceful-termination/0.log"
Jul 13 20:20:04.466812 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:04.466780    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9" event={"ID":"71e4daec-773e-4190-bf03-cdcdb7b4d313","Type":"ContainerStarted","Data":"f8328a01c17c0cea00416b6c2adc2899ed9076576f77ccd7d23160f848593a16"}
Jul 13 20:20:04.483769 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:04.483725    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-n6nx9" podStartSLOduration=1.885121171 podStartE2EDuration="4.483710111s" totalImagesPullingTime="2.59858894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:04.483323017 +0000 UTC m=+49.835931888" watchObservedRunningTime="2026-07-13 20:20:04.483710111 +0000 UTC m=+49.836318983"
Jul 13 20:20:06.669508 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:06.669471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:06.669508 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:06.669515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:06.669898 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:06.669587    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:20:06.669898 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:06.669608    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:06.669898 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:06.669644    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:14.669629598 +0000 UTC m=+60.022238418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : configmap "kube-rbac-proxy" not found
Jul 13 20:20:06.669898 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:06.669670    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:14.669663847 +0000 UTC m=+60.022272668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:07.373497 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:07.373456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:20:07.373780 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:07.373608    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:20:07.373780 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:07.373672    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls podName:6dd0dafe-9870-40d4-b882-724ab09f3395 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:23.373655401 +0000 UTC m=+68.726264242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls") pod "dns-default-lrsdq" (UID: "6dd0dafe-9870-40d4-b882-724ab09f3395") : secret "dns-default-metrics-tls" not found
Jul 13 20:20:14.731154 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:14.731121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:14.731625 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:14.731164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:14.731625 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:14.731256    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:14.731625 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:20:14.731309    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls podName:4e3d19be-0c48-4ee6-8ec5-26059fc9e318 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:30.731295702 +0000 UTC m=+76.083904536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbtzv" (UID: "4e3d19be-0c48-4ee6-8ec5-26059fc9e318") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:20:14.731732 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:14.731624    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:15.411725 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:15.411688    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwl74"
Jul 13 20:20:20.873848 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.873812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:20:20.873848 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.873852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:20:20.879543 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.879519    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 20:20:20.879672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.879599    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 20:20:20.886327 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.886306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1347ce06-e218-4be5-9dd2-5db94f75ff35-metrics-certs\") pod \"network-metrics-daemon-nrn7d\" (UID: \"1347ce06-e218-4be5-9dd2-5db94f75ff35\") " pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:20:20.886404 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.886348    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/636333a8-2370-4e1f-acc9-4ad49e62fcfc-original-pull-secret\") pod \"global-pull-secret-syncer-hqxqb\" (UID: \"636333a8-2370-4e1f-acc9-4ad49e62fcfc\") " pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:20:20.943498 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.943472    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6z4f\""
Jul 13 20:20:20.944521 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.944503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqxqb"
Jul 13 20:20:20.950411 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.950387    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nrn7d"
Jul 13 20:20:20.974663 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.974634    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:20:20.977907 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.977888    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 20:20:20.988708 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:20.988684    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 20:20:21.001251 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.001196    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvn9k\" (UniqueName: \"kubernetes.io/projected/9556dff5-b188-4ff9-aaf2-dc315a1ba735-kube-api-access-qvn9k\") pod \"network-check-target-r557g\" (UID: \"9556dff5-b188-4ff9-aaf2-dc315a1ba735\") " pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:20:21.080454 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.080421    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hqxqb"]
Jul 13 20:20:21.083677 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:21.083645    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636333a8_2370_4e1f_acc9_4ad49e62fcfc.slice/crio-7ee328dce79290b1d552eb42513dc685c09f503d3f574fe7719f678a609499ff WatchSource:0}: Error finding container 7ee328dce79290b1d552eb42513dc685c09f503d3f574fe7719f678a609499ff: Status 404 returned error can't find the container with id 7ee328dce79290b1d552eb42513dc685c09f503d3f574fe7719f678a609499ff
Jul 13 20:20:21.099279 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.099259    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nrn7d"]
Jul 13 20:20:21.101356 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:21.101335    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1347ce06_e218_4be5_9dd2_5db94f75ff35.slice/crio-ab050b9449a50678b6bcb728dcec7632203f8c15bb8c259c78d43f679f698d90 WatchSource:0}: Error finding container ab050b9449a50678b6bcb728dcec7632203f8c15bb8c259c78d43f679f698d90: Status 404 returned error can't find the container with id ab050b9449a50678b6bcb728dcec7632203f8c15bb8c259c78d43f679f698d90
Jul 13 20:20:21.238044 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.238016    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-899ms\""
Jul 13 20:20:21.245882 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.245867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:20:21.371959 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.371927    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r557g"]
Jul 13 20:20:21.375050 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:21.375028    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9556dff5_b188_4ff9_aaf2_dc315a1ba735.slice/crio-1b1264f2b12977c8987560d1acc02b02e1bd1ff980b0c73df95ac17af2424652 WatchSource:0}: Error finding container 1b1264f2b12977c8987560d1acc02b02e1bd1ff980b0c73df95ac17af2424652: Status 404 returned error can't find the container with id 1b1264f2b12977c8987560d1acc02b02e1bd1ff980b0c73df95ac17af2424652
Jul 13 20:20:21.510754 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.510669    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hqxqb" event={"ID":"636333a8-2370-4e1f-acc9-4ad49e62fcfc","Type":"ContainerStarted","Data":"7ee328dce79290b1d552eb42513dc685c09f503d3f574fe7719f678a609499ff"}
Jul 13 20:20:21.511769 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.511748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r557g" event={"ID":"9556dff5-b188-4ff9-aaf2-dc315a1ba735","Type":"ContainerStarted","Data":"1b1264f2b12977c8987560d1acc02b02e1bd1ff980b0c73df95ac17af2424652"}
Jul 13 20:20:21.512781 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:21.512760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nrn7d" event={"ID":"1347ce06-e218-4be5-9dd2-5db94f75ff35","Type":"ContainerStarted","Data":"ab050b9449a50678b6bcb728dcec7632203f8c15bb8c259c78d43f679f698d90"}
Jul 13 20:20:22.519502 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.519464    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nrn7d" event={"ID":"1347ce06-e218-4be5-9dd2-5db94f75ff35","Type":"ContainerStarted","Data":"8157f701e915cf59b1b59167b67ee3161b48457d630851f3323a9bc5bb6ec594"}
Jul 13 20:20:22.519975 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.519563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nrn7d" event={"ID":"1347ce06-e218-4be5-9dd2-5db94f75ff35","Type":"ContainerStarted","Data":"42172e3e24bf659930e66caa2093e9f38faa5e02d65d820386dd581f24d217fe"}
Jul 13 20:20:22.543860 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.543639    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nrn7d" podStartSLOduration=66.587455556 podStartE2EDuration="1m7.54362134s" totalImagesPullingTime="956.165784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:22.543416053 +0000 UTC m=+67.896024896" watchObservedRunningTime="2026-07-13 20:20:22.54362134 +0000 UTC m=+67.896230176"
Jul 13 20:20:22.643898 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.643861    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-stzmg"]
Jul 13 20:20:22.647320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.646958    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.655486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.655464    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 13 20:20:22.657029 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.657009    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-69rgf\""
Jul 13 20:20:22.658054 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.658036    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 13 20:20:22.668408 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.668387    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-stzmg"]
Jul 13 20:20:22.690070 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.689835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dab114e9-37f1-4dae-a99d-9a8533cddcca-nginx-conf\") pod \"networking-console-plugin-6cf856758c-stzmg\" (UID: \"dab114e9-37f1-4dae-a99d-9a8533cddcca\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.690070 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.689955    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dab114e9-37f1-4dae-a99d-9a8533cddcca-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-stzmg\" (UID: \"dab114e9-37f1-4dae-a99d-9a8533cddcca\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.759809 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.759769    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"]
Jul 13 20:20:22.762968 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.762908    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"]
Jul 13 20:20:22.763124 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.763079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:22.767110 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.767085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.770420 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.770349    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 20:20:22.770420 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.770410    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 20:20:22.773394 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.772772    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 20:20:22.773394 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.773030    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 13 20:20:22.773394 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.773236    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 13 20:20:22.773589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.773491    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 13 20:20:22.774169 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.774149    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 13 20:20:22.774286 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.774251    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q2hhz\""
Jul 13 20:20:22.774917 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.774896    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 13 20:20:22.789036 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.789015    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"]
Jul 13 20:20:22.790842 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.790820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dab114e9-37f1-4dae-a99d-9a8533cddcca-nginx-conf\") pod \"networking-console-plugin-6cf856758c-stzmg\" (UID: \"dab114e9-37f1-4dae-a99d-9a8533cddcca\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.790934 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.790913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dab114e9-37f1-4dae-a99d-9a8533cddcca-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-stzmg\" (UID: \"dab114e9-37f1-4dae-a99d-9a8533cddcca\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.791812 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.791785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dab114e9-37f1-4dae-a99d-9a8533cddcca-nginx-conf\") pod \"networking-console-plugin-6cf856758c-stzmg\" (UID: \"dab114e9-37f1-4dae-a99d-9a8533cddcca\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.797521 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.797472    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"]
Jul 13 20:20:22.798566 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.798518    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dab114e9-37f1-4dae-a99d-9a8533cddcca-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-stzmg\" (UID: \"dab114e9-37f1-4dae-a99d-9a8533cddcca\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.891736 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdbn\" (UniqueName: \"kubernetes.io/projected/386070d8-de24-4e69-bc3c-0f83e991607a-kube-api-access-5bdbn\") pod \"managed-serviceaccount-addon-agent-86f49d7956-9k9sz\" (UID: \"386070d8-de24-4e69-bc3c-0f83e991607a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:22.891924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.891924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/300a6edb-80d1-42ff-ab95-7838910834dc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.891924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-ca\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.891924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.891924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dl9\" (UniqueName: \"kubernetes.io/projected/300a6edb-80d1-42ff-ab95-7838910834dc-kube-api-access-d2dl9\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.891924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891909    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-hub\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.892227 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.891952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/386070d8-de24-4e69-bc3c-0f83e991607a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86f49d7956-9k9sz\" (UID: \"386070d8-de24-4e69-bc3c-0f83e991607a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:22.959031 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.958967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg"
Jul 13 20:20:22.993187 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/386070d8-de24-4e69-bc3c-0f83e991607a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86f49d7956-9k9sz\" (UID: \"386070d8-de24-4e69-bc3c-0f83e991607a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:22.993341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdbn\" (UniqueName: \"kubernetes.io/projected/386070d8-de24-4e69-bc3c-0f83e991607a-kube-api-access-5bdbn\") pod \"managed-serviceaccount-addon-agent-86f49d7956-9k9sz\" (UID: \"386070d8-de24-4e69-bc3c-0f83e991607a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:22.993341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.993341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/300a6edb-80d1-42ff-ab95-7838910834dc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.993341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-ca\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.993535 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.993535 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dl9\" (UniqueName: \"kubernetes.io/projected/300a6edb-80d1-42ff-ab95-7838910834dc-kube-api-access-d2dl9\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.993535 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.993410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-hub\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.994371 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.994323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/300a6edb-80d1-42ff-ab95-7838910834dc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.998680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.998618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:22.999930 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:22.999865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/386070d8-de24-4e69-bc3c-0f83e991607a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86f49d7956-9k9sz\" (UID: \"386070d8-de24-4e69-bc3c-0f83e991607a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:23.000562 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.000521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:23.000644 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.000595    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-ca\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:23.001902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.001874    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/300a6edb-80d1-42ff-ab95-7838910834dc-hub\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:23.019277 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.019224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dl9\" (UniqueName: \"kubernetes.io/projected/300a6edb-80d1-42ff-ab95-7838910834dc-kube-api-access-d2dl9\") pod \"cluster-proxy-proxy-agent-5ddd6c94f4-qztzr\" (UID: \"300a6edb-80d1-42ff-ab95-7838910834dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:23.022752 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.022673    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdbn\" (UniqueName: \"kubernetes.io/projected/386070d8-de24-4e69-bc3c-0f83e991607a-kube-api-access-5bdbn\") pod \"managed-serviceaccount-addon-agent-86f49d7956-9k9sz\" (UID: \"386070d8-de24-4e69-bc3c-0f83e991607a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:23.084703 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.084406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"
Jul 13 20:20:23.091363 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.091122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"
Jul 13 20:20:23.140812 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.140697    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-stzmg"]
Jul 13 20:20:23.144018 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:23.143957    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab114e9_37f1_4dae_a99d_9a8533cddcca.slice/crio-627b4fa6f64b226819abd564792cba1716bca7d78d81e8d110db1a8192c4d04c WatchSource:0}: Error finding container 627b4fa6f64b226819abd564792cba1716bca7d78d81e8d110db1a8192c4d04c: Status 404 returned error can't find the container with id 627b4fa6f64b226819abd564792cba1716bca7d78d81e8d110db1a8192c4d04c
Jul 13 20:20:23.264189 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.264152    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz"]
Jul 13 20:20:23.267892 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:23.267862    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod386070d8_de24_4e69_bc3c_0f83e991607a.slice/crio-00e78e996aeb6acf99dda57b152daaeb30f6f87668def21bdffa5d22e8d1a1e9 WatchSource:0}: Error finding container 00e78e996aeb6acf99dda57b152daaeb30f6f87668def21bdffa5d22e8d1a1e9: Status 404 returned error can't find the container with id 00e78e996aeb6acf99dda57b152daaeb30f6f87668def21bdffa5d22e8d1a1e9
Jul 13 20:20:23.283387 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.283357    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr"]
Jul 13 20:20:23.396705 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.396580    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:20:23.400279 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.400226    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd0dafe-9870-40d4-b882-724ab09f3395-metrics-tls\") pod \"dns-default-lrsdq\" (UID: \"6dd0dafe-9870-40d4-b882-724ab09f3395\") " pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:20:23.525730 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.525633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz" event={"ID":"386070d8-de24-4e69-bc3c-0f83e991607a","Type":"ContainerStarted","Data":"00e78e996aeb6acf99dda57b152daaeb30f6f87668def21bdffa5d22e8d1a1e9"}
Jul 13 20:20:23.526954 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.526924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg" event={"ID":"dab114e9-37f1-4dae-a99d-9a8533cddcca","Type":"ContainerStarted","Data":"627b4fa6f64b226819abd564792cba1716bca7d78d81e8d110db1a8192c4d04c"}
Jul 13 20:20:23.529029 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.528445    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr" event={"ID":"300a6edb-80d1-42ff-ab95-7838910834dc","Type":"ContainerStarted","Data":"8eb216f74ad53695be8ceba3695629637ea79b54c6d803d612042bab342ac940"}
Jul 13 20:20:23.684658 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.684624    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t5qs6\""
Jul 13 20:20:23.692307 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.692279    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:20:23.880231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:23.880153    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lrsdq"]
Jul 13 20:20:23.885298 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:23.885263    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd0dafe_9870_40d4_b882_724ab09f3395.slice/crio-fbb71f1fc3d6efd607c1a9bf096a12de0161d01e67a0398633c0ad487c53dabe WatchSource:0}: Error finding container fbb71f1fc3d6efd607c1a9bf096a12de0161d01e67a0398633c0ad487c53dabe: Status 404 returned error can't find the container with id fbb71f1fc3d6efd607c1a9bf096a12de0161d01e67a0398633c0ad487c53dabe
Jul 13 20:20:24.534575 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:24.534535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrsdq" event={"ID":"6dd0dafe-9870-40d4-b882-724ab09f3395","Type":"ContainerStarted","Data":"fbb71f1fc3d6efd607c1a9bf096a12de0161d01e67a0398633c0ad487c53dabe"}
Jul 13 20:20:30.557789 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.557751    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrsdq" event={"ID":"6dd0dafe-9870-40d4-b882-724ab09f3395","Type":"ContainerStarted","Data":"3406ac1b3b6cc9f48cb31eb596f681b46bda4957c33d57b1e3cd7b6f7f7831b0"}
Jul 13 20:20:30.558256 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.557805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrsdq" event={"ID":"6dd0dafe-9870-40d4-b882-724ab09f3395","Type":"ContainerStarted","Data":"976ab2c5eb1ef9410bc64da9a5b70ebad901c75314af2f8507874e618f518a2c"}
Jul 13 20:20:30.558256 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.557837    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:20:30.559347 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.559323    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r557g" event={"ID":"9556dff5-b188-4ff9-aaf2-dc315a1ba735","Type":"ContainerStarted","Data":"ae50b75f3ec30a86cf0c8adf050504a53167a3b69b52426de9b8ca49db646f98"}
Jul 13 20:20:30.559446 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.559379    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:20:30.561384 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.561359    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr" event={"ID":"300a6edb-80d1-42ff-ab95-7838910834dc","Type":"ContainerStarted","Data":"052ea829ed94c3c302650158590eba3db7e135adb969d9fd681eb20da3fcb470"}
Jul 13 20:20:30.561472 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.561399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr" event={"ID":"300a6edb-80d1-42ff-ab95-7838910834dc","Type":"ContainerStarted","Data":"971eaf0f19606381fd594f0e49efbcf04258b81c02b8e95213da3a89836fcebe"}
Jul 13 20:20:30.561472 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.561418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr" event={"ID":"300a6edb-80d1-42ff-ab95-7838910834dc","Type":"ContainerStarted","Data":"18232748a0512b3b20bab999d2e227c858f305c405c232f5557048bfa5dfe322"}
Jul 13 20:20:30.562824 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.562806    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hqxqb" event={"ID":"636333a8-2370-4e1f-acc9-4ad49e62fcfc","Type":"ContainerStarted","Data":"505d0ab4cfa800fab9c08bffa1e5901f93245c5515355d8babd4c8594ac0a8b5"}
Jul 13 20:20:30.564180 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.564156    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz" event={"ID":"386070d8-de24-4e69-bc3c-0f83e991607a","Type":"ContainerStarted","Data":"6daafdc113df0b12ce78c33179f78cd93aca8aa17a485d55772e8c5fbdd70a71"}
Jul 13 20:20:30.565523 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.565506    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg" event={"ID":"dab114e9-37f1-4dae-a99d-9a8533cddcca","Type":"ContainerStarted","Data":"86139efa5bbfbf8bae613c241f355ec0bf6ffaaf3e49e17b6b0eb2c99b0946d7"}
Jul 13 20:20:30.578945 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.578908    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lrsdq" podStartSLOduration=33.32890773 podStartE2EDuration="39.578895918s" totalImagesPullingTime="6.249988188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:30.577182951 +0000 UTC m=+75.929791807" watchObservedRunningTime="2026-07-13 20:20:30.578895918 +0000 UTC m=+75.931504761"
Jul 13 20:20:30.599787 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.599719    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-stzmg" podStartSLOduration=1.662422026 podStartE2EDuration="8.599706409s" totalImagesPullingTime="6.937284383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:30.59876642 +0000 UTC m=+75.951375266" watchObservedRunningTime="2026-07-13 20:20:30.599706409 +0000 UTC m=+75.952315246"
Jul 13 20:20:30.615800 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.615753    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r557g" podStartSLOduration=66.913175734 podStartE2EDuration="1m15.615736417s" totalImagesPullingTime="8.702560683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:30.61556709 +0000 UTC m=+75.968175934" watchObservedRunningTime="2026-07-13 20:20:30.615736417 +0000 UTC m=+75.968345261"
Jul 13 20:20:30.662939 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.662840    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86f49d7956-9k9sz" podStartSLOduration=1.817731623 podStartE2EDuration="8.6628213s" totalImagesPullingTime="6.845089677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:30.637249838 +0000 UTC m=+75.989858692" watchObservedRunningTime="2026-07-13 20:20:30.6628213 +0000 UTC m=+76.015430144"
Jul 13 20:20:30.663124 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.662935    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hqxqb" podStartSLOduration=66.668606491 podStartE2EDuration="1m15.662927287s" totalImagesPullingTime="8.994320796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:30.662135101 +0000 UTC m=+76.014743945" watchObservedRunningTime="2026-07-13 20:20:30.662927287 +0000 UTC m=+76.015536133"
Jul 13 20:20:30.714449 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.714400    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5ddd6c94f4-qztzr" podStartSLOduration=1.921770599 podStartE2EDuration="8.714384378s" totalImagesPullingTime="6.792613779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:30.713917724 +0000 UTC m=+76.066526566" watchObservedRunningTime="2026-07-13 20:20:30.714384378 +0000 UTC m=+76.066993221"
Jul 13 20:20:30.757303 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.757272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:30.759616 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.759598    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e3d19be-0c48-4ee6-8ec5-26059fc9e318-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbtzv\" (UID: \"4e3d19be-0c48-4ee6-8ec5-26059fc9e318\") " pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:30.972133 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.972102    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nlmqs\""
Jul 13 20:20:30.979669 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:30.979646    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kbtzv"
Jul 13 20:20:31.110160 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:31.110127    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kbtzv"]
Jul 13 20:20:31.113423 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:31.113398    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3d19be_0c48_4ee6_8ec5_26059fc9e318.slice/crio-b2d1edacebe36b3e547bd8608c7d027e9356613f3303bab074296c883614693b WatchSource:0}: Error finding container b2d1edacebe36b3e547bd8608c7d027e9356613f3303bab074296c883614693b: Status 404 returned error can't find the container with id b2d1edacebe36b3e547bd8608c7d027e9356613f3303bab074296c883614693b
Jul 13 20:20:31.569903 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:31.569868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbtzv" event={"ID":"4e3d19be-0c48-4ee6-8ec5-26059fc9e318","Type":"ContainerStarted","Data":"b2d1edacebe36b3e547bd8608c7d027e9356613f3303bab074296c883614693b"}
Jul 13 20:20:35.585626 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:35.585503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbtzv" event={"ID":"4e3d19be-0c48-4ee6-8ec5-26059fc9e318","Type":"ContainerStarted","Data":"23ad9eb879f0c8a958ccfb8ce9a8da31dfec589afcd90d24cb9a12e460d0d984"}
Jul 13 20:20:36.590728 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:36.590640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbtzv" event={"ID":"4e3d19be-0c48-4ee6-8ec5-26059fc9e318","Type":"ContainerStarted","Data":"b56ec1ebea26b355431ac61d79db70cf910edc210c754a219e4f56843a41347d"}
Jul 13 20:20:39.603182 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:39.603145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbtzv" event={"ID":"4e3d19be-0c48-4ee6-8ec5-26059fc9e318","Type":"ContainerStarted","Data":"4dcb58d79309548e99f8ce23d20fa1a8bf939f5b746894300dd82d7a855b0bcc"}
Jul 13 20:20:39.623742 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:39.623692    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kbtzv" podStartSLOduration=34.098467532 podStartE2EDuration="41.623677328s" totalImagesPullingTime="7.525209796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:39.623456364 +0000 UTC m=+84.976065207" watchObservedRunningTime="2026-07-13 20:20:39.623677328 +0000 UTC m=+84.976286170"
Jul 13 20:20:40.572591 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:40.572562    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lrsdq"
Jul 13 20:20:47.729476 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.727240    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fnm95"]
Jul 13 20:20:47.731770 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.731746    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.734578 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.734553    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 20:20:47.735603 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.735578    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 20:20:47.735696 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.735620    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pks4v\""
Jul 13 20:20:47.735696 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.735675    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 20:20:47.735814 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.735621    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 20:20:47.735814 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.735810    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 20:20:47.735932 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.735815    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 20:20:47.876973 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.876918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-sys\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6dw\" (UniqueName: \"kubernetes.io/projected/15f8d971-bcd4-4779-b5af-c13f76c13dd9-kube-api-access-hs6dw\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-root\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15f8d971-bcd4-4779-b5af-c13f76c13dd9-metrics-client-ca\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-textfile\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877384 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-tls\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.877384 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.877209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-wtmp\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.977913 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.977877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-textfile\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.977913 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.977910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-tls\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.977926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-wtmp\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.977959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-sys\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6dw\" (UniqueName: \"kubernetes.io/projected/15f8d971-bcd4-4779-b5af-c13f76c13dd9-kube-api-access-hs6dw\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-root\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978126    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-wtmp\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15f8d971-bcd4-4779-b5af-c13f76c13dd9-metrics-client-ca\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978568 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-sys\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978568 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978199    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15f8d971-bcd4-4779-b5af-c13f76c13dd9-root\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978568 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978267    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-textfile\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978763 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.978817 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.978798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15f8d971-bcd4-4779-b5af-c13f76c13dd9-metrics-client-ca\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.980664 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.980607    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.980735 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.980678    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15f8d971-bcd4-4779-b5af-c13f76c13dd9-node-exporter-tls\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:47.987757 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:47.987737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6dw\" (UniqueName: \"kubernetes.io/projected/15f8d971-bcd4-4779-b5af-c13f76c13dd9-kube-api-access-hs6dw\") pod \"node-exporter-fnm95\" (UID: \"15f8d971-bcd4-4779-b5af-c13f76c13dd9\") " pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:48.041260 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:48.041237    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fnm95"
Jul 13 20:20:48.052763 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:20:48.052739    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f8d971_bcd4_4779_b5af_c13f76c13dd9.slice/crio-582a1677239f9c3af15c1402fbd0b9a974ff0eee138abd6e5411a02344e3983c WatchSource:0}: Error finding container 582a1677239f9c3af15c1402fbd0b9a974ff0eee138abd6e5411a02344e3983c: Status 404 returned error can't find the container with id 582a1677239f9c3af15c1402fbd0b9a974ff0eee138abd6e5411a02344e3983c
Jul 13 20:20:48.634820 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:48.634775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnm95" event={"ID":"15f8d971-bcd4-4779-b5af-c13f76c13dd9","Type":"ContainerStarted","Data":"582a1677239f9c3af15c1402fbd0b9a974ff0eee138abd6e5411a02344e3983c"}
Jul 13 20:20:49.640323 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:49.640287    2575 generic.go:356] "Generic (PLEG): container finished" podID="15f8d971-bcd4-4779-b5af-c13f76c13dd9" containerID="3a9dfd57aafa398d5dc1bdb239e11f1057bb8fd2cc4f5d7771abb135c7cdec98" exitCode=0
Jul 13 20:20:49.640729 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:49.640351    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnm95" event={"ID":"15f8d971-bcd4-4779-b5af-c13f76c13dd9","Type":"ContainerDied","Data":"3a9dfd57aafa398d5dc1bdb239e11f1057bb8fd2cc4f5d7771abb135c7cdec98"}
Jul 13 20:20:50.648811 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:50.648778    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnm95" event={"ID":"15f8d971-bcd4-4779-b5af-c13f76c13dd9","Type":"ContainerStarted","Data":"6c95eb92d6af63432b282c6a36ef958b4fdb5287b43d6843d5033ef664564365"}
Jul 13 20:20:50.649200 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:50.648828    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fnm95" event={"ID":"15f8d971-bcd4-4779-b5af-c13f76c13dd9","Type":"ContainerStarted","Data":"1ee9e9453c0f9e84e85ec4eb292eb71359e424cfd430fd62864dcf1d3713beed"}
Jul 13 20:20:50.673845 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:20:50.673798    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fnm95" podStartSLOduration=2.754509571 podStartE2EDuration="3.673783189s" totalImagesPullingTime="849.777015ms" totalInitContainerRuntime="69.496603ms" isStatefulPod=true podCreationTimestamp="2026-07-13 20:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:50.672164014 +0000 UTC m=+96.024772857" watchObservedRunningTime="2026-07-13 20:20:50.673783189 +0000 UTC m=+96.026392032"
Jul 13 20:21:01.573030 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:21:01.572979    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r557g"
Jul 13 20:24:14.981822 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:14.981791    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:24:14.982381 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:14.982022    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:24:14.989024 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:14.989002    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 20:24:23.542676 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.542643    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"]
Jul 13 20:24:23.545873 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.545857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.551180 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.551146    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 13 20:24:23.551318 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.551233    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8898f\""
Jul 13 20:24:23.551416 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.551396    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 13 20:24:23.552427 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.552288    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 13 20:24:23.563497 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.563475    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"]
Jul 13 20:24:23.630432 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.630398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e5d70d8b-f273-4e2a-847c-cb065a90a256-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-wnghg\" (UID: \"e5d70d8b-f273-4e2a-847c-cb065a90a256\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.630594 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.630441    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pfcs\" (UniqueName: \"kubernetes.io/projected/e5d70d8b-f273-4e2a-847c-cb065a90a256-kube-api-access-8pfcs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-wnghg\" (UID: \"e5d70d8b-f273-4e2a-847c-cb065a90a256\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.731520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.731489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pfcs\" (UniqueName: \"kubernetes.io/projected/e5d70d8b-f273-4e2a-847c-cb065a90a256-kube-api-access-8pfcs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-wnghg\" (UID: \"e5d70d8b-f273-4e2a-847c-cb065a90a256\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.731691 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.731573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e5d70d8b-f273-4e2a-847c-cb065a90a256-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-wnghg\" (UID: \"e5d70d8b-f273-4e2a-847c-cb065a90a256\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.734029 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.734004    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e5d70d8b-f273-4e2a-847c-cb065a90a256-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-wnghg\" (UID: \"e5d70d8b-f273-4e2a-847c-cb065a90a256\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.741416 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.741395    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pfcs\" (UniqueName: \"kubernetes.io/projected/e5d70d8b-f273-4e2a-847c-cb065a90a256-kube-api-access-8pfcs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-wnghg\" (UID: \"e5d70d8b-f273-4e2a-847c-cb065a90a256\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.855601 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.855515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:23.987337 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.987316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"]
Jul 13 20:24:23.989861 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:24:23.989830    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d70d8b_f273_4e2a_847c_cb065a90a256.slice/crio-33e587a0038fbb94abb249c657d4bd12f56977ae4a10f24c3cade76d15ff967c WatchSource:0}: Error finding container 33e587a0038fbb94abb249c657d4bd12f56977ae4a10f24c3cade76d15ff967c: Status 404 returned error can't find the container with id 33e587a0038fbb94abb249c657d4bd12f56977ae4a10f24c3cade76d15ff967c
Jul 13 20:24:23.991670 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:23.991652    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:24:24.354817 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:24.354784    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg" event={"ID":"e5d70d8b-f273-4e2a-847c-cb065a90a256","Type":"ContainerStarted","Data":"33e587a0038fbb94abb249c657d4bd12f56977ae4a10f24c3cade76d15ff967c"}
Jul 13 20:24:27.368049 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:27.368013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg" event={"ID":"e5d70d8b-f273-4e2a-847c-cb065a90a256","Type":"ContainerStarted","Data":"f0d6b9d3ce6591b19cd4b65aaa104ae93ec0cf202bc8b1e1d4bcad28099f4555"}
Jul 13 20:24:27.368416 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:27.368117    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:24:27.396811 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:27.396759    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg" podStartSLOduration=1.48774301 podStartE2EDuration="4.396740122s" totalImagesPullingTime="2.908997112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:24:27.395636677 +0000 UTC m=+312.748245531" watchObservedRunningTime="2026-07-13 20:24:27.396740122 +0000 UTC m=+312.749348964"
Jul 13 20:24:48.374212 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:24:48.374176    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-wnghg"
Jul 13 20:25:37.805394 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.805356    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68b649c569-frllw"]
Jul 13 20:25:37.808232 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.808216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:37.812142 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.812124    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-b48pc\""
Jul 13 20:25:37.812623 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.812604    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 13 20:25:37.813270 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.813254    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 13 20:25:37.813270 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.813266    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 13 20:25:37.822356 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.822323    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68b649c569-frllw"]
Jul 13 20:25:37.857842 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.857818    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-z64s4"]
Jul 13 20:25:37.860714 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.860699    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:37.868008 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:25:37.867957    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"seaweedfs-64568bcd49-z64s4\" is forbidden: User \"system:node:ip-10-0-128-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-128-233.ec2.internal' and this object" podUID="cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155" pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:37.868580 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.868566    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 13 20:25:37.870298 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.870283    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j98tn\""
Jul 13 20:25:37.882312 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.882290    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-z64s4"]
Jul 13 20:25:37.955704 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.955667    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adefc1ed-828b-40fb-a786-cd8e9a4afea4-cert\") pod \"kserve-controller-manager-68b649c569-frllw\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") " pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:37.955704 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.955706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155-data\") pod \"seaweedfs-64568bcd49-z64s4\" (UID: \"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155\") " pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:37.955918 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.955729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwp5\" (UniqueName: \"kubernetes.io/projected/cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155-kube-api-access-lwwp5\") pod \"seaweedfs-64568bcd49-z64s4\" (UID: \"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155\") " pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:37.955918 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:37.955816    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjm4t\" (UniqueName: \"kubernetes.io/projected/adefc1ed-828b-40fb-a786-cd8e9a4afea4-kube-api-access-wjm4t\") pod \"kserve-controller-manager-68b649c569-frllw\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") " pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:38.056836 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.056748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adefc1ed-828b-40fb-a786-cd8e9a4afea4-cert\") pod \"kserve-controller-manager-68b649c569-frllw\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") " pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:38.056836 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.056785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155-data\") pod \"seaweedfs-64568bcd49-z64s4\" (UID: \"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155\") " pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:38.056836 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.056807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwp5\" (UniqueName: \"kubernetes.io/projected/cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155-kube-api-access-lwwp5\") pod \"seaweedfs-64568bcd49-z64s4\" (UID: \"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155\") " pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:38.056836 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.056834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjm4t\" (UniqueName: \"kubernetes.io/projected/adefc1ed-828b-40fb-a786-cd8e9a4afea4-kube-api-access-wjm4t\") pod \"kserve-controller-manager-68b649c569-frllw\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") " pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:38.057202 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.057164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155-data\") pod \"seaweedfs-64568bcd49-z64s4\" (UID: \"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155\") " pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:38.059493 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.059464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adefc1ed-828b-40fb-a786-cd8e9a4afea4-cert\") pod \"kserve-controller-manager-68b649c569-frllw\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") " pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:38.066675 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.066648    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwp5\" (UniqueName: \"kubernetes.io/projected/cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155-kube-api-access-lwwp5\") pod \"seaweedfs-64568bcd49-z64s4\" (UID: \"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155\") " pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:38.066884 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.066865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjm4t\" (UniqueName: \"kubernetes.io/projected/adefc1ed-828b-40fb-a786-cd8e9a4afea4-kube-api-access-wjm4t\") pod \"kserve-controller-manager-68b649c569-frllw\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") " pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:38.119051 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.119020    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:38.170768 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.170737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:38.256551 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.256520    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68b649c569-frllw"]
Jul 13 20:25:38.261574 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:25:38.261543    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadefc1ed_828b_40fb_a786_cd8e9a4afea4.slice/crio-83519ceb4dd2282db7142c1fb38e498adf8d8981202e70857eca6ee0a2eb3640 WatchSource:0}: Error finding container 83519ceb4dd2282db7142c1fb38e498adf8d8981202e70857eca6ee0a2eb3640: Status 404 returned error can't find the container with id 83519ceb4dd2282db7142c1fb38e498adf8d8981202e70857eca6ee0a2eb3640
Jul 13 20:25:38.308712 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.308506    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-z64s4"]
Jul 13 20:25:38.311161 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:25:38.311113    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac9a4d4_3a0a_4474_b1a4_60b4ec9d3155.slice/crio-6f3949c56f6d9b0f2a16631e890d3b14a57c234783264d38bb441ce9f69eb948 WatchSource:0}: Error finding container 6f3949c56f6d9b0f2a16631e890d3b14a57c234783264d38bb441ce9f69eb948: Status 404 returned error can't find the container with id 6f3949c56f6d9b0f2a16631e890d3b14a57c234783264d38bb441ce9f69eb948
Jul 13 20:25:38.605174 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.605090    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68b649c569-frllw" event={"ID":"adefc1ed-828b-40fb-a786-cd8e9a4afea4","Type":"ContainerStarted","Data":"83519ceb4dd2282db7142c1fb38e498adf8d8981202e70857eca6ee0a2eb3640"}
Jul 13 20:25:38.606366 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:38.606341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-z64s4" event={"ID":"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155","Type":"ContainerStarted","Data":"6f3949c56f6d9b0f2a16631e890d3b14a57c234783264d38bb441ce9f69eb948"}
Jul 13 20:25:42.622846 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:42.622811    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68b649c569-frllw" event={"ID":"adefc1ed-828b-40fb-a786-cd8e9a4afea4","Type":"ContainerStarted","Data":"0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252"}
Jul 13 20:25:42.623296 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:42.622907    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:25:42.624486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:42.624458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-z64s4" event={"ID":"cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155","Type":"ContainerStarted","Data":"46fb7515462946c360caea670b298ba403c34d43e7d21e68de347ce18b6a2e0d"}
Jul 13 20:25:42.624597 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:42.624512    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:25:42.688472 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:42.688420    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68b649c569-frllw" podStartSLOduration=2.138763169 podStartE2EDuration="5.688405535s" totalImagesPullingTime="3.549642366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:25:42.687726855 +0000 UTC m=+388.040335697" watchObservedRunningTime="2026-07-13 20:25:42.688405535 +0000 UTC m=+388.041014378"
Jul 13 20:25:42.718179 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:42.718133    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-z64s4" podStartSLOduration=2.12829283 podStartE2EDuration="5.718118013s" totalImagesPullingTime="3.589825183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:25:42.718024258 +0000 UTC m=+388.070633103" watchObservedRunningTime="2026-07-13 20:25:42.718118013 +0000 UTC m=+388.070726876"
Jul 13 20:25:48.630794 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:25:48.630766    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-z64s4"
Jul 13 20:26:13.634041 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:13.634011    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:26:14.430235 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.430195    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68b649c569-frllw"]
Jul 13 20:26:14.430524 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.430481    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-68b649c569-frllw" podUID="adefc1ed-828b-40fb-a786-cd8e9a4afea4" containerName="manager" containerID="cri-o://0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252" gracePeriod=10
Jul 13 20:26:14.464338 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.464315    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68b649c569-ztbrc"]
Jul 13 20:26:14.469112 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.469096    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.480476 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.480449    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68b649c569-ztbrc"]
Jul 13 20:26:14.504261 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.504237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89811c72-6825-4c59-9b3e-7be3d1319e7e-cert\") pod \"kserve-controller-manager-68b649c569-ztbrc\" (UID: \"89811c72-6825-4c59-9b3e-7be3d1319e7e\") " pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.504383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.504272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psx44\" (UniqueName: \"kubernetes.io/projected/89811c72-6825-4c59-9b3e-7be3d1319e7e-kube-api-access-psx44\") pod \"kserve-controller-manager-68b649c569-ztbrc\" (UID: \"89811c72-6825-4c59-9b3e-7be3d1319e7e\") " pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.605446 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.605419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89811c72-6825-4c59-9b3e-7be3d1319e7e-cert\") pod \"kserve-controller-manager-68b649c569-ztbrc\" (UID: \"89811c72-6825-4c59-9b3e-7be3d1319e7e\") " pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.605561 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.605466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psx44\" (UniqueName: \"kubernetes.io/projected/89811c72-6825-4c59-9b3e-7be3d1319e7e-kube-api-access-psx44\") pod \"kserve-controller-manager-68b649c569-ztbrc\" (UID: \"89811c72-6825-4c59-9b3e-7be3d1319e7e\") " pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.607960 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.607933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89811c72-6825-4c59-9b3e-7be3d1319e7e-cert\") pod \"kserve-controller-manager-68b649c569-ztbrc\" (UID: \"89811c72-6825-4c59-9b3e-7be3d1319e7e\") " pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.615353 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.615326    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psx44\" (UniqueName: \"kubernetes.io/projected/89811c72-6825-4c59-9b3e-7be3d1319e7e-kube-api-access-psx44\") pod \"kserve-controller-manager-68b649c569-ztbrc\" (UID: \"89811c72-6825-4c59-9b3e-7be3d1319e7e\") " pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.702130 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.702100    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:26:14.735035 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.735002    2575 generic.go:356] "Generic (PLEG): container finished" podID="adefc1ed-828b-40fb-a786-cd8e9a4afea4" containerID="0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252" exitCode=0
Jul 13 20:26:14.735216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.735045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68b649c569-frllw" event={"ID":"adefc1ed-828b-40fb-a786-cd8e9a4afea4","Type":"ContainerDied","Data":"0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252"}
Jul 13 20:26:14.735216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.735061    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68b649c569-frllw"
Jul 13 20:26:14.735216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.735099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68b649c569-frllw" event={"ID":"adefc1ed-828b-40fb-a786-cd8e9a4afea4","Type":"ContainerDied","Data":"83519ceb4dd2282db7142c1fb38e498adf8d8981202e70857eca6ee0a2eb3640"}
Jul 13 20:26:14.735216 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.735130    2575 scope.go:117] "RemoveContainer" containerID="0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252"
Jul 13 20:26:14.743847 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.743829    2575 scope.go:117] "RemoveContainer" containerID="0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252"
Jul 13 20:26:14.744186 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:26:14.744165    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252\": container with ID starting with 0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252 not found: ID does not exist" containerID="0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252"
Jul 13 20:26:14.744263 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.744194    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252"} err="failed to get container status \"0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252\": rpc error: code = NotFound desc = could not find container \"0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252\": container with ID starting with 0dc44c6cf83e83f82b54542df6ed88bcd0a2f352bd5e5df52fa5ffd559fb7252 not found: ID does not exist"
Jul 13 20:26:14.807899 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.807871    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adefc1ed-828b-40fb-a786-cd8e9a4afea4-cert\") pod \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") "
Jul 13 20:26:14.808102 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.807909    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjm4t\" (UniqueName: \"kubernetes.io/projected/adefc1ed-828b-40fb-a786-cd8e9a4afea4-kube-api-access-wjm4t\") pod \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\" (UID: \"adefc1ed-828b-40fb-a786-cd8e9a4afea4\") "
Jul 13 20:26:14.810182 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.810158    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adefc1ed-828b-40fb-a786-cd8e9a4afea4-kube-api-access-wjm4t" (OuterVolumeSpecName: "kube-api-access-wjm4t") pod "adefc1ed-828b-40fb-a786-cd8e9a4afea4" (UID: "adefc1ed-828b-40fb-a786-cd8e9a4afea4"). InnerVolumeSpecName "kube-api-access-wjm4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:26:14.810264 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.810176    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adefc1ed-828b-40fb-a786-cd8e9a4afea4-cert" (OuterVolumeSpecName: "cert") pod "adefc1ed-828b-40fb-a786-cd8e9a4afea4" (UID: "adefc1ed-828b-40fb-a786-cd8e9a4afea4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:26:14.817341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.817320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:14.909010 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.908970    2575 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adefc1ed-828b-40fb-a786-cd8e9a4afea4-cert\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:26:14.909141 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.909014    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjm4t\" (UniqueName: \"kubernetes.io/projected/adefc1ed-828b-40fb-a786-cd8e9a4afea4-kube-api-access-wjm4t\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:26:14.974587 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:14.974501    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68b649c569-ztbrc"]
Jul 13 20:26:14.977251 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:26:14.977222    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89811c72_6825_4c59_9b3e_7be3d1319e7e.slice/crio-2de16aeac59e5bcf75c373d5eb89ebd8d68f0f3ed7720959e8a1db0fa8cd6c50 WatchSource:0}: Error finding container 2de16aeac59e5bcf75c373d5eb89ebd8d68f0f3ed7720959e8a1db0fa8cd6c50: Status 404 returned error can't find the container with id 2de16aeac59e5bcf75c373d5eb89ebd8d68f0f3ed7720959e8a1db0fa8cd6c50
Jul 13 20:26:15.056270 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:15.056243    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68b649c569-frllw"]
Jul 13 20:26:15.059509 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:15.059486    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-68b649c569-frllw"]
Jul 13 20:26:15.129069 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:15.129039    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adefc1ed-828b-40fb-a786-cd8e9a4afea4" path="/var/lib/kubelet/pods/adefc1ed-828b-40fb-a786-cd8e9a4afea4/volumes"
Jul 13 20:26:15.740575 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:15.740489    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68b649c569-ztbrc" event={"ID":"89811c72-6825-4c59-9b3e-7be3d1319e7e","Type":"ContainerStarted","Data":"c7c5ea6af04a25e01ed28ef258265063775c41077a43194cc3dcab1aa92c75da"}
Jul 13 20:26:15.740575 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:15.740543    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68b649c569-ztbrc" event={"ID":"89811c72-6825-4c59-9b3e-7be3d1319e7e","Type":"ContainerStarted","Data":"2de16aeac59e5bcf75c373d5eb89ebd8d68f0f3ed7720959e8a1db0fa8cd6c50"}
Jul 13 20:26:15.741006 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:15.740617    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:15.764473 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:15.764425    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68b649c569-ztbrc" podStartSLOduration=1.161487523 podStartE2EDuration="1.764408258s" totalImagesPullingTime="602.920735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:26:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:26:15.761919297 +0000 UTC m=+421.114528140" watchObservedRunningTime="2026-07-13 20:26:15.764408258 +0000 UTC m=+421.117017142"
Jul 13 20:26:46.749534 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:46.749460    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68b649c569-ztbrc"
Jul 13 20:26:47.974418 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.974387    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-9w8qp"]
Jul 13 20:26:47.974855 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.974738    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adefc1ed-828b-40fb-a786-cd8e9a4afea4" containerName="manager"
Jul 13 20:26:47.974855 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.974754    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="adefc1ed-828b-40fb-a786-cd8e9a4afea4" containerName="manager"
Jul 13 20:26:47.974855 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.974836    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="adefc1ed-828b-40fb-a786-cd8e9a4afea4" containerName="manager"
Jul 13 20:26:47.977786 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.977764    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:47.980437 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.980414    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 13 20:26:47.980566 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.980464    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-xpjss\""
Jul 13 20:26:47.992072 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:47.992051    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-9w8qp"]
Jul 13 20:26:48.050763 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.050734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7af263c0-d8a2-4efd-8c92-d885b7f43cb2-cert\") pod \"odh-model-controller-84b54b7f97-9w8qp\" (UID: \"7af263c0-d8a2-4efd-8c92-d885b7f43cb2\") " pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:48.050923 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.050776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l869l\" (UniqueName: \"kubernetes.io/projected/7af263c0-d8a2-4efd-8c92-d885b7f43cb2-kube-api-access-l869l\") pod \"odh-model-controller-84b54b7f97-9w8qp\" (UID: \"7af263c0-d8a2-4efd-8c92-d885b7f43cb2\") " pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:48.151446 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.151412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7af263c0-d8a2-4efd-8c92-d885b7f43cb2-cert\") pod \"odh-model-controller-84b54b7f97-9w8qp\" (UID: \"7af263c0-d8a2-4efd-8c92-d885b7f43cb2\") " pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:48.151607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.151451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l869l\" (UniqueName: \"kubernetes.io/projected/7af263c0-d8a2-4efd-8c92-d885b7f43cb2-kube-api-access-l869l\") pod \"odh-model-controller-84b54b7f97-9w8qp\" (UID: \"7af263c0-d8a2-4efd-8c92-d885b7f43cb2\") " pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:48.153855 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.153835    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7af263c0-d8a2-4efd-8c92-d885b7f43cb2-cert\") pod \"odh-model-controller-84b54b7f97-9w8qp\" (UID: \"7af263c0-d8a2-4efd-8c92-d885b7f43cb2\") " pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:48.160960 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.160939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l869l\" (UniqueName: \"kubernetes.io/projected/7af263c0-d8a2-4efd-8c92-d885b7f43cb2-kube-api-access-l869l\") pod \"odh-model-controller-84b54b7f97-9w8qp\" (UID: \"7af263c0-d8a2-4efd-8c92-d885b7f43cb2\") " pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:48.288298 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.288212    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:26:48.426772 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.426738    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-9w8qp"]
Jul 13 20:26:48.431677 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:26:48.431648    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af263c0_d8a2_4efd_8c92_d885b7f43cb2.slice/crio-e7b9096cb8d8a019c5216709817a7be94ac6761a15d810115d358345ed63e28c WatchSource:0}: Error finding container e7b9096cb8d8a019c5216709817a7be94ac6761a15d810115d358345ed63e28c: Status 404 returned error can't find the container with id e7b9096cb8d8a019c5216709817a7be94ac6761a15d810115d358345ed63e28c
Jul 13 20:26:48.856842 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:48.856808    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-9w8qp" event={"ID":"7af263c0-d8a2-4efd-8c92-d885b7f43cb2","Type":"ContainerStarted","Data":"e7b9096cb8d8a019c5216709817a7be94ac6761a15d810115d358345ed63e28c"}
Jul 13 20:26:51.871637 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:51.871599    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-9w8qp" event={"ID":"7af263c0-d8a2-4efd-8c92-d885b7f43cb2","Type":"ContainerStarted","Data":"840a3fffb197882ba3d3b61bd92b8c175b2c26ea57b2edd6ac47a0dfa6763ca3"}
Jul 13 20:26:51.872018 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:26:51.871754    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:27:02.877334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:27:02.877304    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-9w8qp"
Jul 13 20:27:02.899832 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:27:02.899782    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-9w8qp" podStartSLOduration=13.121493568 podStartE2EDuration="15.899768686s" totalImagesPullingTime="2.778275118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:26:51.895716854 +0000 UTC m=+457.248325697" watchObservedRunningTime="2026-07-13 20:27:02.899768686 +0000 UTC m=+468.252377528"
Jul 13 20:28:34.036400 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.036360    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"]
Jul 13 20:28:34.039596 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.039571    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.042974 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.042955    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 20:28:34.043261 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.043243    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 20:28:34.043338 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.043316    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cbd18-predictor-serving-cert\""
Jul 13 20:28:34.044281 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.044261    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cbd18-kube-rbac-proxy-sar-config\""
Jul 13 20:28:34.044375 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.044263    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kvq55\""
Jul 13 20:28:34.052765 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.052747    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"]
Jul 13 20:28:34.125397 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.125367    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d77q\" (UniqueName: \"kubernetes.io/projected/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-api-access-8d77q\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.125571 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.125409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.125571 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.125495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-proxy-tls\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.210309 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.210271    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"]
Jul 13 20:28:34.214005 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.213966    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.218026 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.218007    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cbd18-predictor-serving-cert\""
Jul 13 20:28:34.218152 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.218036    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cbd18-kube-rbac-proxy-sar-config\""
Jul 13 20:28:34.224217 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.224191    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"]
Jul 13 20:28:34.226108 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.226072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d77q\" (UniqueName: \"kubernetes.io/projected/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-api-access-8d77q\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.226211 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.226127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.226211 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.226186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-proxy-tls\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.227201 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.227174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.229180 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.229154    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-proxy-tls\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.240905 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.240884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d77q\" (UniqueName: \"kubernetes.io/projected/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-api-access-8d77q\") pod \"success-200-isvc-cbd18-predictor-749f5f5477-xmgml\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") " pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.326966 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.326870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7b4\" (UniqueName: \"kubernetes.io/projected/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-api-access-vd7b4\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.326966 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.326938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.327269 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.327013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.350114 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.350078    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:34.430585 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.429352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.430585 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.429414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.430585 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.429463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7b4\" (UniqueName: \"kubernetes.io/projected/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-api-access-vd7b4\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.430585 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:28:34.430111    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-serving-cert: secret "error-404-isvc-cbd18-predictor-serving-cert" not found
Jul 13 20:28:34.430585 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:28:34.430178    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls podName:d9c637a5-3e0d-49a6-8d80-aa5e33c8173e nodeName:}" failed. No retries permitted until 2026-07-13 20:28:34.930157709 +0000 UTC m=+560.282766550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls") pod "error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" (UID: "d9c637a5-3e0d-49a6-8d80-aa5e33c8173e") : secret "error-404-isvc-cbd18-predictor-serving-cert" not found
Jul 13 20:28:34.430585 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.430538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.446344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.446280    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7b4\" (UniqueName: \"kubernetes.io/projected/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-api-access-vd7b4\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.517270 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.517227    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"]
Jul 13 20:28:34.520368 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:28:34.520337    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d4fc3c_cd56_4763_a3bb_aab8d81f2292.slice/crio-b3744639b7e596a4ae642ab1cbd60dd46d27e2e620efe5c134eb8b6ea7d59b6e WatchSource:0}: Error finding container b3744639b7e596a4ae642ab1cbd60dd46d27e2e620efe5c134eb8b6ea7d59b6e: Status 404 returned error can't find the container with id b3744639b7e596a4ae642ab1cbd60dd46d27e2e620efe5c134eb8b6ea7d59b6e
Jul 13 20:28:34.934310 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.934276    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:34.936714 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:34.936694    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls\") pod \"error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") " pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:35.128869 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:35.128518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:35.245202 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:35.245107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" event={"ID":"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292","Type":"ContainerStarted","Data":"b3744639b7e596a4ae642ab1cbd60dd46d27e2e620efe5c134eb8b6ea7d59b6e"}
Jul 13 20:28:35.327220 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:35.327103    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"]
Jul 13 20:28:35.333736 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:28:35.333705    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c637a5_3e0d_49a6_8d80_aa5e33c8173e.slice/crio-6ab332219e7969cfc45e4630515f196fa1d815becf663485ffd73b8c49938be7 WatchSource:0}: Error finding container 6ab332219e7969cfc45e4630515f196fa1d815becf663485ffd73b8c49938be7: Status 404 returned error can't find the container with id 6ab332219e7969cfc45e4630515f196fa1d815becf663485ffd73b8c49938be7
Jul 13 20:28:36.270043 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:36.270003    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" event={"ID":"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e","Type":"ContainerStarted","Data":"6ab332219e7969cfc45e4630515f196fa1d815becf663485ffd73b8c49938be7"}
Jul 13 20:28:50.346378 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:50.346343    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" event={"ID":"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e","Type":"ContainerStarted","Data":"e5f7250143c5ba61fea70ed1939881603f42af48a271907bd836017b10a51c8e"}
Jul 13 20:28:50.347848 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:50.347817    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" event={"ID":"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292","Type":"ContainerStarted","Data":"a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8"}
Jul 13 20:28:53.363749 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.363709    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" event={"ID":"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292","Type":"ContainerStarted","Data":"0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436"}
Jul 13 20:28:53.364267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.363790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:53.364267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.363820    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:53.365349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.365314    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 13 20:28:53.365596 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.365573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" event={"ID":"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e","Type":"ContainerStarted","Data":"42bd934bd59536d1d6ed2969a90cc748922c09f1988bee179f82302c5b1b3051"}
Jul 13 20:28:53.365712 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.365699    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:53.387087 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.387040    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podStartSLOduration=1.029966513 podStartE2EDuration="19.387026211s" totalImagesPullingTime="18.357059698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:28:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:28:53.384334446 +0000 UTC m=+578.736943290" watchObservedRunningTime="2026-07-13 20:28:53.387026211 +0000 UTC m=+578.739635053"
Jul 13 20:28:53.407133 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:53.406976    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podStartSLOduration=1.891742687 podStartE2EDuration="19.406959594s" totalImagesPullingTime="17.515216907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:28:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:28:53.405700761 +0000 UTC m=+578.758309606" watchObservedRunningTime="2026-07-13 20:28:53.406959594 +0000 UTC m=+578.759568437"
Jul 13 20:28:54.369652 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:54.369615    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:28:54.369652 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:54.369638    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 13 20:28:54.370752 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:54.370731    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 13 20:28:55.373695 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:55.373653    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 13 20:28:59.375164 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:59.375136    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:28:59.375641 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:28:59.375613    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 13 20:29:00.378338 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:00.378308    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:29:00.378763 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:00.378742    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 13 20:29:09.375744 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:09.375710    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 13 20:29:10.378747 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:10.378697    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 13 20:29:15.012791 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:15.012759    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:29:15.013250 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:15.013233    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:29:19.376088 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:19.376041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 13 20:29:20.379257 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:20.379222    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 13 20:29:29.375711 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:29.375670    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 13 20:29:30.379656 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:30.379617    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 13 20:29:39.377188 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:39.377102    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:29:40.380189 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:29:40.380155    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:30:04.148721 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.148692    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"]
Jul 13 20:30:04.149218 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.148962    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container" containerID="cri-o://e5f7250143c5ba61fea70ed1939881603f42af48a271907bd836017b10a51c8e" gracePeriod=30
Jul 13 20:30:04.149218 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.149017    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kube-rbac-proxy" containerID="cri-o://42bd934bd59536d1d6ed2969a90cc748922c09f1988bee179f82302c5b1b3051" gracePeriod=30
Jul 13 20:30:04.212503 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.212470    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"]
Jul 13 20:30:04.212729 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.212698    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container" containerID="cri-o://a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8" gracePeriod=30
Jul 13 20:30:04.212844 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.212755    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kube-rbac-proxy" containerID="cri-o://0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436" gracePeriod=30
Jul 13 20:30:04.291346 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.291305    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"]
Jul 13 20:30:04.299085 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.299056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.301844 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.301812    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-91263-predictor-serving-cert\""
Jul 13 20:30:04.301977 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.301831    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-91263-kube-rbac-proxy-sar-config\""
Jul 13 20:30:04.306645 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.306367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"]
Jul 13 20:30:04.370827 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.370793    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 13 20:30:04.421411 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.421337    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"]
Jul 13 20:30:04.424567 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.424545    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.427266 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.427241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.427379 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.427297    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be140f10-ffd1-4681-9ca0-4f895724bd3f-proxy-tls\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.427444 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.427411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcbk\" (UniqueName: \"kubernetes.io/projected/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-api-access-hmcbk\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.429087 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.429057    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-91263-predictor-serving-cert\""
Jul 13 20:30:04.429087 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.429073    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-91263-kube-rbac-proxy-sar-config\""
Jul 13 20:30:04.435231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.435212    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"]
Jul 13 20:30:04.528830 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.528778    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.529032 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.528847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be140f10-ffd1-4681-9ca0-4f895724bd3f-proxy-tls\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.529032 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.528935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcbk\" (UniqueName: \"kubernetes.io/projected/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-api-access-hmcbk\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.529032 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.528976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfmg\" (UniqueName: \"kubernetes.io/projected/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-api-access-hwfmg\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.529199 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.529141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.529253 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.529207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.529816 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.529795    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.531779 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.531756    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be140f10-ffd1-4681-9ca0-4f895724bd3f-proxy-tls\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.541298 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.541275    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcbk\" (UniqueName: \"kubernetes.io/projected/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-api-access-hmcbk\") pod \"success-200-isvc-91263-predictor-867c654bd7-l96fd\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") " pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.610891 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.610850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:04.628617 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.628590    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerID="0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436" exitCode=2
Jul 13 20:30:04.628735 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.628672    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" event={"ID":"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292","Type":"ContainerDied","Data":"0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436"}
Jul 13 20:30:04.629579 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.629547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.629683 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.629589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.629683 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.629648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfmg\" (UniqueName: \"kubernetes.io/projected/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-api-access-hwfmg\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.629795 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:30:04.629772    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-91263-predictor-serving-cert: secret "error-404-isvc-91263-predictor-serving-cert" not found
Jul 13 20:30:04.629868 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:30:04.629848    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls podName:16e207be-60cb-4faa-95e9-f6929e0d536b nodeName:}" failed. No retries permitted until 2026-07-13 20:30:05.129826386 +0000 UTC m=+650.482435224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls") pod "error-404-isvc-91263-predictor-5f479d8546-d2nfn" (UID: "16e207be-60cb-4faa-95e9-f6929e0d536b") : secret "error-404-isvc-91263-predictor-serving-cert" not found
Jul 13 20:30:04.630267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.630247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.630658 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.630634    2575 generic.go:356] "Generic (PLEG): container finished" podID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerID="42bd934bd59536d1d6ed2969a90cc748922c09f1988bee179f82302c5b1b3051" exitCode=2
Jul 13 20:30:04.630731 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.630682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" event={"ID":"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e","Type":"ContainerDied","Data":"42bd934bd59536d1d6ed2969a90cc748922c09f1988bee179f82302c5b1b3051"}
Jul 13 20:30:04.644625 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.644594    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfmg\" (UniqueName: \"kubernetes.io/projected/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-api-access-hwfmg\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:04.750904 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:04.750871    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"]
Jul 13 20:30:04.755076 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:30:04.755048    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe140f10_ffd1_4681_9ca0_4f895724bd3f.slice/crio-8be992ccade8d4e816b68697f177c11e5295426d3572d665cac421a0bf4e2d97 WatchSource:0}: Error finding container 8be992ccade8d4e816b68697f177c11e5295426d3572d665cac421a0bf4e2d97: Status 404 returned error can't find the container with id 8be992ccade8d4e816b68697f177c11e5295426d3572d665cac421a0bf4e2d97
Jul 13 20:30:05.133754 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.133722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:05.136224 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.136206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls\") pod \"error-404-isvc-91263-predictor-5f479d8546-d2nfn\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") " pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:05.335651 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.335613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:05.374082 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.374029    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 13 20:30:05.472311 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.472279    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"]
Jul 13 20:30:05.476976 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:30:05.476945    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e207be_60cb_4faa_95e9_f6929e0d536b.slice/crio-2f361869fa3adb0118d704670d782e14e7d735dc78a86e1e68d5cd56bd90e3a9 WatchSource:0}: Error finding container 2f361869fa3adb0118d704670d782e14e7d735dc78a86e1e68d5cd56bd90e3a9: Status 404 returned error can't find the container with id 2f361869fa3adb0118d704670d782e14e7d735dc78a86e1e68d5cd56bd90e3a9
Jul 13 20:30:05.637776 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.637683    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" event={"ID":"16e207be-60cb-4faa-95e9-f6929e0d536b","Type":"ContainerStarted","Data":"e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730"}
Jul 13 20:30:05.637776 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.637749    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" event={"ID":"16e207be-60cb-4faa-95e9-f6929e0d536b","Type":"ContainerStarted","Data":"a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d"}
Jul 13 20:30:05.638001 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.637785    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:05.638001 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.637892    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:05.638001 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.637942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" event={"ID":"16e207be-60cb-4faa-95e9-f6929e0d536b","Type":"ContainerStarted","Data":"2f361869fa3adb0118d704670d782e14e7d735dc78a86e1e68d5cd56bd90e3a9"}
Jul 13 20:30:05.639328 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.639301    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 20:30:05.639645 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.639626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" event={"ID":"be140f10-ffd1-4681-9ca0-4f895724bd3f","Type":"ContainerStarted","Data":"b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46"}
Jul 13 20:30:05.639699 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.639674    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" event={"ID":"be140f10-ffd1-4681-9ca0-4f895724bd3f","Type":"ContainerStarted","Data":"fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9"}
Jul 13 20:30:05.639730 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.639700    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:05.639730 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.639718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" event={"ID":"be140f10-ffd1-4681-9ca0-4f895724bd3f","Type":"ContainerStarted","Data":"8be992ccade8d4e816b68697f177c11e5295426d3572d665cac421a0bf4e2d97"}
Jul 13 20:30:05.661365 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.661309    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podStartSLOduration=1.661292784 podStartE2EDuration="1.661292784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:30:05.661126787 +0000 UTC m=+651.013735631" watchObservedRunningTime="2026-07-13 20:30:05.661292784 +0000 UTC m=+651.013901625"
Jul 13 20:30:05.686589 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:05.686488    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podStartSLOduration=1.686465281 podStartE2EDuration="1.686465281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:30:05.683948417 +0000 UTC m=+651.036557271" watchObservedRunningTime="2026-07-13 20:30:05.686465281 +0000 UTC m=+651.039074124"
Jul 13 20:30:06.644014 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:06.643955    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:06.644482 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:06.644120    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 20:30:06.645344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:06.645320    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 20:30:07.558848 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.558822    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:30:07.649921 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.649888    2575 generic.go:356] "Generic (PLEG): container finished" podID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerID="e5f7250143c5ba61fea70ed1939881603f42af48a271907bd836017b10a51c8e" exitCode=0
Jul 13 20:30:07.650320 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.649966    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" event={"ID":"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e","Type":"ContainerDied","Data":"e5f7250143c5ba61fea70ed1939881603f42af48a271907bd836017b10a51c8e"}
Jul 13 20:30:07.654021 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.653978    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerID="a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8" exitCode=0
Jul 13 20:30:07.654197 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.654048    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"
Jul 13 20:30:07.654197 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.654064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" event={"ID":"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292","Type":"ContainerDied","Data":"a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8"}
Jul 13 20:30:07.654197 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.654122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml" event={"ID":"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292","Type":"ContainerDied","Data":"b3744639b7e596a4ae642ab1cbd60dd46d27e2e620efe5c134eb8b6ea7d59b6e"}
Jul 13 20:30:07.654197 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.654159    2575 scope.go:117] "RemoveContainer" containerID="0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436"
Jul 13 20:30:07.654618 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.654586    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 20:30:07.662224 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.662207    2575 scope.go:117] "RemoveContainer" containerID="a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8"
Jul 13 20:30:07.668971 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.668958    2575 scope.go:117] "RemoveContainer" containerID="0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436"
Jul 13 20:30:07.669218 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:30:07.669201    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436\": container with ID starting with 0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436 not found: ID does not exist" containerID="0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436"
Jul 13 20:30:07.669269 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.669227    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436"} err="failed to get container status \"0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436\": rpc error: code = NotFound desc = could not find container \"0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436\": container with ID starting with 0c3191e13769d0004c1d39cdaa18b088986a88477aabd2c92b7ae1c1d0ee6436 not found: ID does not exist"
Jul 13 20:30:07.669269 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.669244    2575 scope.go:117] "RemoveContainer" containerID="a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8"
Jul 13 20:30:07.669444 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:30:07.669429    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8\": container with ID starting with a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8 not found: ID does not exist" containerID="a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8"
Jul 13 20:30:07.669502 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.669447    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8"} err="failed to get container status \"a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8\": rpc error: code = NotFound desc = could not find container \"a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8\": container with ID starting with a5a1bf4f907cd9002d96a24faf5c24614c612a7a59bfdf81ae2b9d22ac6b87e8 not found: ID does not exist"
Jul 13 20:30:07.753800 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.753765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-proxy-tls\") pod \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") "
Jul 13 20:30:07.753959 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.753849    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d77q\" (UniqueName: \"kubernetes.io/projected/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-api-access-8d77q\") pod \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") "
Jul 13 20:30:07.753959 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.753875    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-rbac-proxy-sar-config\") pod \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\" (UID: \"b7d4fc3c-cd56-4763-a3bb-aab8d81f2292\") "
Jul 13 20:30:07.754304 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.754267    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" (UID: "b7d4fc3c-cd56-4763-a3bb-aab8d81f2292"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:30:07.756327 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.756296    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-api-access-8d77q" (OuterVolumeSpecName: "kube-api-access-8d77q") pod "b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" (UID: "b7d4fc3c-cd56-4763-a3bb-aab8d81f2292"). InnerVolumeSpecName "kube-api-access-8d77q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:30:07.756327 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.756318    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" (UID: "b7d4fc3c-cd56-4763-a3bb-aab8d81f2292"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:30:07.855182 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.855156    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:30:07.855182 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.855181    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d77q\" (UniqueName: \"kubernetes.io/projected/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-api-access-8d77q\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:30:07.855348 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.855193    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:30:07.889844 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.889809    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:30:07.977534 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.977504    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"]
Jul 13 20:30:07.981958 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:07.981933    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbd18-predictor-749f5f5477-xmgml"]
Jul 13 20:30:08.056796 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.056758    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-rbac-proxy-sar-config\") pod \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") "
Jul 13 20:30:08.056976 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.056873    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd7b4\" (UniqueName: \"kubernetes.io/projected/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-api-access-vd7b4\") pod \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") "
Jul 13 20:30:08.056976 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.056902    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls\") pod \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\" (UID: \"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e\") "
Jul 13 20:30:08.057167 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.057144    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" (UID: "d9c637a5-3e0d-49a6-8d80-aa5e33c8173e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:30:08.059096 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.059064    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" (UID: "d9c637a5-3e0d-49a6-8d80-aa5e33c8173e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:30:08.059193 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.059125    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-api-access-vd7b4" (OuterVolumeSpecName: "kube-api-access-vd7b4") pod "d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" (UID: "d9c637a5-3e0d-49a6-8d80-aa5e33c8173e"). InnerVolumeSpecName "kube-api-access-vd7b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:30:08.158334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.158252    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7b4\" (UniqueName: \"kubernetes.io/projected/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-api-access-vd7b4\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:30:08.158334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.158281    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:30:08.158334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.158290    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:30:08.659576 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.659545    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"
Jul 13 20:30:08.659576 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.659554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88" event={"ID":"d9c637a5-3e0d-49a6-8d80-aa5e33c8173e","Type":"ContainerDied","Data":"6ab332219e7969cfc45e4630515f196fa1d815becf663485ffd73b8c49938be7"}
Jul 13 20:30:08.660082 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.659625    2575 scope.go:117] "RemoveContainer" containerID="42bd934bd59536d1d6ed2969a90cc748922c09f1988bee179f82302c5b1b3051"
Jul 13 20:30:08.668199 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.668182    2575 scope.go:117] "RemoveContainer" containerID="e5f7250143c5ba61fea70ed1939881603f42af48a271907bd836017b10a51c8e"
Jul 13 20:30:08.688671 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.688644    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"]
Jul 13 20:30:08.693398 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:08.693371    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cbd18-predictor-69b7b99dcb-lxn88"]
Jul 13 20:30:09.129365 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:09.129332    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" path="/var/lib/kubelet/pods/b7d4fc3c-cd56-4763-a3bb-aab8d81f2292/volumes"
Jul 13 20:30:09.129742 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:09.129728    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" path="/var/lib/kubelet/pods/d9c637a5-3e0d-49a6-8d80-aa5e33c8173e/volumes"
Jul 13 20:30:11.648921 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:11.648892    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:11.649444 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:11.649416    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 20:30:12.658962 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:12.658932    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:30:12.659481 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:12.659455    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 20:30:21.649835 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:21.649789    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 20:30:22.659642 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:22.659604    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 20:30:31.649755 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:31.649711    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 20:30:32.659655 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:32.659621    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 20:30:41.650349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:41.650306    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 20:30:42.660316 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:42.660278    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 20:30:51.649956 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:51.649924    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:30:52.660147 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:30:52.660115    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:31:04.170858 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.170816    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"]
Jul 13 20:31:04.171344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171270    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kube-rbac-proxy"
Jul 13 20:31:04.171344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171289    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kube-rbac-proxy"
Jul 13 20:31:04.171344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171303    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container"
Jul 13 20:31:04.171344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171310    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container"
Jul 13 20:31:04.171344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171322    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container"
Jul 13 20:31:04.171344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171330    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container"
Jul 13 20:31:04.171344 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171342    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kube-rbac-proxy"
Jul 13 20:31:04.171672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171351    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kube-rbac-proxy"
Jul 13 20:31:04.171672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171432    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kube-rbac-proxy"
Jul 13 20:31:04.171672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171443    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kube-rbac-proxy"
Jul 13 20:31:04.171672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171452    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d4fc3c-cd56-4763-a3bb-aab8d81f2292" containerName="kserve-container"
Jul 13 20:31:04.171672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.171462    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c637a5-3e0d-49a6-8d80-aa5e33c8173e" containerName="kserve-container"
Jul 13 20:31:04.174720 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.174698    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.177417 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.177398    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-43619-kube-rbac-proxy-sar-config\""
Jul 13 20:31:04.177538 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.177521    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-43619-predictor-serving-cert\""
Jul 13 20:31:04.186968 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.186942    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"]
Jul 13 20:31:04.260878 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.260836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f18dc863-1999-46d0-aa30-37b4c9151890-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.261070 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.260891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x2k\" (UniqueName: \"kubernetes.io/projected/f18dc863-1999-46d0-aa30-37b4c9151890-kube-api-access-74x2k\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.261246 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.261210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.278922 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.278889    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"]
Jul 13 20:31:04.282372 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.282351    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.285153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.285135    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43619-kube-rbac-proxy-sar-config\""
Jul 13 20:31:04.285267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.285139    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43619-predictor-serving-cert\""
Jul 13 20:31:04.295139 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.295105    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"]
Jul 13 20:31:04.362598 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.362567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f18dc863-1999-46d0-aa30-37b4c9151890-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.362598 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.362608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x2k\" (UniqueName: \"kubernetes.io/projected/f18dc863-1999-46d0-aa30-37b4c9151890-kube-api-access-74x2k\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.362836 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.362731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79k6\" (UniqueName: \"kubernetes.io/projected/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-api-access-m79k6\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.362836 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.362812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.362944 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.362881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.362944 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.362931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.363082 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:31:04.363035    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-43619-predictor-serving-cert: secret "success-200-isvc-43619-predictor-serving-cert" not found
Jul 13 20:31:04.363141 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:31:04.363115    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls podName:f18dc863-1999-46d0-aa30-37b4c9151890 nodeName:}" failed. No retries permitted until 2026-07-13 20:31:04.863094519 +0000 UTC m=+710.215703363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls") pod "success-200-isvc-43619-predictor-857f7657f5-b28dc" (UID: "f18dc863-1999-46d0-aa30-37b4c9151890") : secret "success-200-isvc-43619-predictor-serving-cert" not found
Jul 13 20:31:04.363391 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.363372    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f18dc863-1999-46d0-aa30-37b4c9151890-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.373855 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.373792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x2k\" (UniqueName: \"kubernetes.io/projected/f18dc863-1999-46d0-aa30-37b4c9151890-kube-api-access-74x2k\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.464362 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.464326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79k6\" (UniqueName: \"kubernetes.io/projected/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-api-access-m79k6\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.464538 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.464393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.464538 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.464456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.464672 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:31:04.464583    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-43619-predictor-serving-cert: secret "error-404-isvc-43619-predictor-serving-cert" not found
Jul 13 20:31:04.464672 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:31:04.464672    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls podName:35b0ae30-4d80-4bb6-b7e9-c071481aba95 nodeName:}" failed. No retries permitted until 2026-07-13 20:31:04.964654779 +0000 UTC m=+710.317263607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls") pod "error-404-isvc-43619-predictor-8695456798-vdxlr" (UID: "35b0ae30-4d80-4bb6-b7e9-c071481aba95") : secret "error-404-isvc-43619-predictor-serving-cert" not found
Jul 13 20:31:04.465131 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.465109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.479146 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.479120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79k6\" (UniqueName: \"kubernetes.io/projected/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-api-access-m79k6\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.868680 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.868590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.871060 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.871034    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls\") pod \"success-200-isvc-43619-predictor-857f7657f5-b28dc\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") " pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:04.969868 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.969834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:04.972459 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:04.972432    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls\") pod \"error-404-isvc-43619-predictor-8695456798-vdxlr\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") " pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:05.085526 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.085498    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:05.192508 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.192482    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:05.226685 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.226648    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"]
Jul 13 20:31:05.231094 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:31:05.231066    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18dc863_1999_46d0_aa30_37b4c9151890.slice/crio-766a783344d48c179e4be9c34ffdcf30786d2736ac62d4c7a6c1e76832077c0c WatchSource:0}: Error finding container 766a783344d48c179e4be9c34ffdcf30786d2736ac62d4c7a6c1e76832077c0c: Status 404 returned error can't find the container with id 766a783344d48c179e4be9c34ffdcf30786d2736ac62d4c7a6c1e76832077c0c
Jul 13 20:31:05.331056 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.330958    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"]
Jul 13 20:31:05.333764 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:31:05.333742    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b0ae30_4d80_4bb6_b7e9_c071481aba95.slice/crio-00645550921afd31e61a11acc7fb0dbc3822f94cc2647f16f070ad59236fe590 WatchSource:0}: Error finding container 00645550921afd31e61a11acc7fb0dbc3822f94cc2647f16f070ad59236fe590: Status 404 returned error can't find the container with id 00645550921afd31e61a11acc7fb0dbc3822f94cc2647f16f070ad59236fe590
Jul 13 20:31:05.873095 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.873056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" event={"ID":"35b0ae30-4d80-4bb6-b7e9-c071481aba95","Type":"ContainerStarted","Data":"53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c"}
Jul 13 20:31:05.873275 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.873124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" event={"ID":"35b0ae30-4d80-4bb6-b7e9-c071481aba95","Type":"ContainerStarted","Data":"6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f"}
Jul 13 20:31:05.873275 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.873155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" event={"ID":"35b0ae30-4d80-4bb6-b7e9-c071481aba95","Type":"ContainerStarted","Data":"00645550921afd31e61a11acc7fb0dbc3822f94cc2647f16f070ad59236fe590"}
Jul 13 20:31:05.873275 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.873184    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:05.877485 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.877451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" event={"ID":"f18dc863-1999-46d0-aa30-37b4c9151890","Type":"ContainerStarted","Data":"3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681"}
Jul 13 20:31:05.877629 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.877514    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" event={"ID":"f18dc863-1999-46d0-aa30-37b4c9151890","Type":"ContainerStarted","Data":"daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56"}
Jul 13 20:31:05.877629 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.877538    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" event={"ID":"f18dc863-1999-46d0-aa30-37b4c9151890","Type":"ContainerStarted","Data":"766a783344d48c179e4be9c34ffdcf30786d2736ac62d4c7a6c1e76832077c0c"}
Jul 13 20:31:05.877629 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.877564    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:05.895473 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.895430    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podStartSLOduration=1.89541306 podStartE2EDuration="1.89541306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:31:05.895143081 +0000 UTC m=+711.247751923" watchObservedRunningTime="2026-07-13 20:31:05.89541306 +0000 UTC m=+711.248021902"
Jul 13 20:31:05.914842 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:05.914800    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podStartSLOduration=1.914786366 podStartE2EDuration="1.914786366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:31:05.913635649 +0000 UTC m=+711.266244505" watchObservedRunningTime="2026-07-13 20:31:05.914786366 +0000 UTC m=+711.267395209"
Jul 13 20:31:06.882724 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:06.882682    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:06.883189 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:06.882873    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:06.883720 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:06.883689    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 20:31:06.883818 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:06.883737    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 20:31:07.887665 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:07.887622    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 20:31:07.888111 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:07.887757    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 20:31:12.892340 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:12.892286    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:12.892943 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:12.892918    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:31:12.893106 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:12.893015    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 20:31:12.893425 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:12.893397    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 20:31:22.893750 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:22.893706    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 20:31:22.893750 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:22.893738    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 20:31:32.893419 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:32.893376    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 20:31:32.893419 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:32.893394    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 20:31:42.893560 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:42.893519    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 20:31:42.893959 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:42.893520    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 20:31:52.893162 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:52.893128    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:31:52.893978 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:31:52.893957    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:34:15.040547 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:34:15.040514    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:34:15.042418 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:34:15.042397    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:39:15.062743 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:15.062707    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:39:15.066325 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:15.066297    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:39:19.213812 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.213777    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"]
Jul 13 20:39:19.214240 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.214021    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container" containerID="cri-o://fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9" gracePeriod=30
Jul 13 20:39:19.214240 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.214090    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kube-rbac-proxy" containerID="cri-o://b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46" gracePeriod=30
Jul 13 20:39:19.296323 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.296292    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"]
Jul 13 20:39:19.299672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.299648    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.303274 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.303246    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-02ff4-predictor-serving-cert\""
Jul 13 20:39:19.303709 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.303686    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-02ff4-kube-rbac-proxy-sar-config\""
Jul 13 20:39:19.316788 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.316759    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"]
Jul 13 20:39:19.368603 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.368566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.368758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.368614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-proxy-tls\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.368758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.368636    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7z7\" (UniqueName: \"kubernetes.io/projected/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-api-access-gs7z7\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.386157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.386122    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"]
Jul 13 20:39:19.389816 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.389793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.392677 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.392652    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-02ff4-kube-rbac-proxy-sar-config\""
Jul 13 20:39:19.393137 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.393118    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-02ff4-predictor-serving-cert\""
Jul 13 20:39:19.401382 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.401360    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"]
Jul 13 20:39:19.466822 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.466746    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"]
Jul 13 20:39:19.469590 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.469560    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.469734 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.469611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2scr\" (UniqueName: \"kubernetes.io/projected/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-api-access-n2scr\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.469734 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.469670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.469734 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.469712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.469894 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.469744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-proxy-tls\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.469894 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.469761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7z7\" (UniqueName: \"kubernetes.io/projected/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-api-access-gs7z7\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.470430 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.470408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.472370 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.472344    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-proxy-tls\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.482113 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.482088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7z7\" (UniqueName: \"kubernetes.io/projected/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-api-access-gs7z7\") pod \"success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") " pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.571089 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.571048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.571271 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.571109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2scr\" (UniqueName: \"kubernetes.io/projected/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-api-access-n2scr\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.571271 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.571140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.571271 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:39:19.571205    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-serving-cert: secret "error-404-isvc-02ff4-predictor-serving-cert" not found
Jul 13 20:39:19.571426 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:39:19.571281    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls podName:45429bb1-d02d-41de-815c-9aab4b42fb9f nodeName:}" failed. No retries permitted until 2026-07-13 20:39:20.071262268 +0000 UTC m=+1205.423871124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls") pod "error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" (UID: "45429bb1-d02d-41de-815c-9aab4b42fb9f") : secret "error-404-isvc-02ff4-predictor-serving-cert" not found
Jul 13 20:39:19.571801 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.571781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.583563 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.583540    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2scr\" (UniqueName: \"kubernetes.io/projected/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-api-access-n2scr\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:19.612247 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.612218    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:19.753224 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.753141    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"]
Jul 13 20:39:19.755935 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:39:19.755902    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9d18b8_9dac_4e40_950d_488bc8b8c49f.slice/crio-278e251d8102d99b369c1149878e406857b325f3bd79aa97b51902d8860de2b6 WatchSource:0}: Error finding container 278e251d8102d99b369c1149878e406857b325f3bd79aa97b51902d8860de2b6: Status 404 returned error can't find the container with id 278e251d8102d99b369c1149878e406857b325f3bd79aa97b51902d8860de2b6
Jul 13 20:39:19.869917 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.869889    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" event={"ID":"1e9d18b8-9dac-4e40-950d-488bc8b8c49f","Type":"ContainerStarted","Data":"dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3"}
Jul 13 20:39:19.870050 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.869957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" event={"ID":"1e9d18b8-9dac-4e40-950d-488bc8b8c49f","Type":"ContainerStarted","Data":"278e251d8102d99b369c1149878e406857b325f3bd79aa97b51902d8860de2b6"}
Jul 13 20:39:19.871741 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.871722    2575 generic.go:356] "Generic (PLEG): container finished" podID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerID="b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46" exitCode=2
Jul 13 20:39:19.871831 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.871791    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" event={"ID":"be140f10-ffd1-4681-9ca0-4f895724bd3f","Type":"ContainerDied","Data":"b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46"}
Jul 13 20:39:19.871944 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.871922    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" containerID="cri-o://a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d" gracePeriod=30
Jul 13 20:39:19.872045 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:19.872015    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kube-rbac-proxy" containerID="cri-o://e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730" gracePeriod=30
Jul 13 20:39:20.076699 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.076610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:20.079107 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.079083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls\") pod \"error-404-isvc-02ff4-predictor-7748bf4588-2k6pk\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") " pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:20.302374 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.302333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:20.433162 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.433125    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"]
Jul 13 20:39:20.437744 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:39:20.437718    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45429bb1_d02d_41de_815c_9aab4b42fb9f.slice/crio-7632c3c0e1274be522508cc14763ad6325f714140d81b1c6607b2884ea543643 WatchSource:0}: Error finding container 7632c3c0e1274be522508cc14763ad6325f714140d81b1c6607b2884ea543643: Status 404 returned error can't find the container with id 7632c3c0e1274be522508cc14763ad6325f714140d81b1c6607b2884ea543643
Jul 13 20:39:20.878099 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.878060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" event={"ID":"45429bb1-d02d-41de-815c-9aab4b42fb9f","Type":"ContainerStarted","Data":"d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6"}
Jul 13 20:39:20.878306 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.878139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" event={"ID":"45429bb1-d02d-41de-815c-9aab4b42fb9f","Type":"ContainerStarted","Data":"cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0"}
Jul 13 20:39:20.878306 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.878169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" event={"ID":"45429bb1-d02d-41de-815c-9aab4b42fb9f","Type":"ContainerStarted","Data":"7632c3c0e1274be522508cc14763ad6325f714140d81b1c6607b2884ea543643"}
Jul 13 20:39:20.878306 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.878200    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:20.879689 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.879665    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" event={"ID":"1e9d18b8-9dac-4e40-950d-488bc8b8c49f","Type":"ContainerStarted","Data":"08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483"}
Jul 13 20:39:20.879810 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.879720    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:20.879810 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.879786    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:20.881297 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.881273    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 20:39:20.881502 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.881482    2575 generic.go:356] "Generic (PLEG): container finished" podID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerID="e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730" exitCode=2
Jul 13 20:39:20.881553 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.881530    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" event={"ID":"16e207be-60cb-4faa-95e9-f6929e0d536b","Type":"ContainerDied","Data":"e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730"}
Jul 13 20:39:20.903064 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.903009    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podStartSLOduration=1.902972336 podStartE2EDuration="1.902972336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:39:20.902305721 +0000 UTC m=+1206.254914564" watchObservedRunningTime="2026-07-13 20:39:20.902972336 +0000 UTC m=+1206.255581179"
Jul 13 20:39:20.925193 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:20.925147    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podStartSLOduration=1.9251308059999999 podStartE2EDuration="1.925130806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:39:20.923545721 +0000 UTC m=+1206.276154578" watchObservedRunningTime="2026-07-13 20:39:20.925130806 +0000 UTC m=+1206.277739653"
Jul 13 20:39:21.645109 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:21.645068    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 13 20:39:21.649401 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:21.649378    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 20:39:21.886701 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:21.886667    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:21.887039 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:21.886848    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 20:39:21.887640 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:21.887611    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 20:39:22.564621 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.564592    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:39:22.703276 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.703242    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be140f10-ffd1-4681-9ca0-4f895724bd3f-proxy-tls\") pod \"be140f10-ffd1-4681-9ca0-4f895724bd3f\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") "
Jul 13 20:39:22.703768 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.703324    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmcbk\" (UniqueName: \"kubernetes.io/projected/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-api-access-hmcbk\") pod \"be140f10-ffd1-4681-9ca0-4f895724bd3f\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") "
Jul 13 20:39:22.703768 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.703412    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-rbac-proxy-sar-config\") pod \"be140f10-ffd1-4681-9ca0-4f895724bd3f\" (UID: \"be140f10-ffd1-4681-9ca0-4f895724bd3f\") "
Jul 13 20:39:22.703768 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.703732    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be140f10-ffd1-4681-9ca0-4f895724bd3f" (UID: "be140f10-ffd1-4681-9ca0-4f895724bd3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:39:22.705499 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.705481    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be140f10-ffd1-4681-9ca0-4f895724bd3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be140f10-ffd1-4681-9ca0-4f895724bd3f" (UID: "be140f10-ffd1-4681-9ca0-4f895724bd3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:39:22.705598 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.705572    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-api-access-hmcbk" (OuterVolumeSpecName: "kube-api-access-hmcbk") pod "be140f10-ffd1-4681-9ca0-4f895724bd3f" (UID: "be140f10-ffd1-4681-9ca0-4f895724bd3f"). InnerVolumeSpecName "kube-api-access-hmcbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:39:22.804228 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.804192    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcbk\" (UniqueName: \"kubernetes.io/projected/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-api-access-hmcbk\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:39:22.804228 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.804222    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be140f10-ffd1-4681-9ca0-4f895724bd3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:39:22.804228 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.804231    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be140f10-ffd1-4681-9ca0-4f895724bd3f-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:39:22.894389 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.894349    2575 generic.go:356] "Generic (PLEG): container finished" podID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerID="fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9" exitCode=0
Jul 13 20:39:22.894389 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.894393    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"
Jul 13 20:39:22.894607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.894423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" event={"ID":"be140f10-ffd1-4681-9ca0-4f895724bd3f","Type":"ContainerDied","Data":"fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9"}
Jul 13 20:39:22.894607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.894477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd" event={"ID":"be140f10-ffd1-4681-9ca0-4f895724bd3f","Type":"ContainerDied","Data":"8be992ccade8d4e816b68697f177c11e5295426d3572d665cac421a0bf4e2d97"}
Jul 13 20:39:22.894607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.894503    2575 scope.go:117] "RemoveContainer" containerID="b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46"
Jul 13 20:39:22.894874 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.894827    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 20:39:22.903478 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.903458    2575 scope.go:117] "RemoveContainer" containerID="fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9"
Jul 13 20:39:22.910863 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.910842    2575 scope.go:117] "RemoveContainer" containerID="b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46"
Jul 13 20:39:22.911135 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:39:22.911118    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46\": container with ID starting with b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46 not found: ID does not exist" containerID="b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46"
Jul 13 20:39:22.911201 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.911143    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46"} err="failed to get container status \"b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46\": rpc error: code = NotFound desc = could not find container \"b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46\": container with ID starting with b40ed3195b32c567f0562401b0578504a326c98a2d7cca75a5eb554e1460ed46 not found: ID does not exist"
Jul 13 20:39:22.911201 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.911161    2575 scope.go:117] "RemoveContainer" containerID="fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9"
Jul 13 20:39:22.911392 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:39:22.911376    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9\": container with ID starting with fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9 not found: ID does not exist" containerID="fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9"
Jul 13 20:39:22.911428 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.911398    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9"} err="failed to get container status \"fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9\": rpc error: code = NotFound desc = could not find container \"fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9\": container with ID starting with fa64058fa24bec89c94a6ba4ab987ad8fa81869c1f41f58711f44b994205fdc9 not found: ID does not exist"
Jul 13 20:39:22.917670 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.917650    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"]
Jul 13 20:39:22.926291 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:22.926273    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-91263-predictor-867c654bd7-l96fd"]
Jul 13 20:39:23.131394 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.131307    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" path="/var/lib/kubelet/pods/be140f10-ffd1-4681-9ca0-4f895724bd3f/volumes"
Jul 13 20:39:23.824609 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.824586    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:39:23.901123 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.901035    2575 generic.go:356] "Generic (PLEG): container finished" podID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerID="a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d" exitCode=0
Jul 13 20:39:23.901123 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.901084    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"
Jul 13 20:39:23.901305 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.901077    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" event={"ID":"16e207be-60cb-4faa-95e9-f6929e0d536b","Type":"ContainerDied","Data":"a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d"}
Jul 13 20:39:23.901305 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.901227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn" event={"ID":"16e207be-60cb-4faa-95e9-f6929e0d536b","Type":"ContainerDied","Data":"2f361869fa3adb0118d704670d782e14e7d735dc78a86e1e68d5cd56bd90e3a9"}
Jul 13 20:39:23.901305 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.901272    2575 scope.go:117] "RemoveContainer" containerID="e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730"
Jul 13 20:39:23.911479 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.911464    2575 scope.go:117] "RemoveContainer" containerID="a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d"
Jul 13 20:39:23.914520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.914497    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwfmg\" (UniqueName: \"kubernetes.io/projected/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-api-access-hwfmg\") pod \"16e207be-60cb-4faa-95e9-f6929e0d536b\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") "
Jul 13 20:39:23.914618 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.914536    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls\") pod \"16e207be-60cb-4faa-95e9-f6929e0d536b\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") "
Jul 13 20:39:23.914618 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.914602    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-rbac-proxy-sar-config\") pod \"16e207be-60cb-4faa-95e9-f6929e0d536b\" (UID: \"16e207be-60cb-4faa-95e9-f6929e0d536b\") "
Jul 13 20:39:23.914918 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.914894    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16e207be-60cb-4faa-95e9-f6929e0d536b" (UID: "16e207be-60cb-4faa-95e9-f6929e0d536b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:39:23.916781 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.916755    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-api-access-hwfmg" (OuterVolumeSpecName: "kube-api-access-hwfmg") pod "16e207be-60cb-4faa-95e9-f6929e0d536b" (UID: "16e207be-60cb-4faa-95e9-f6929e0d536b"). InnerVolumeSpecName "kube-api-access-hwfmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:39:23.916963 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.916940    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16e207be-60cb-4faa-95e9-f6929e0d536b" (UID: "16e207be-60cb-4faa-95e9-f6929e0d536b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:39:23.919452 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.919421    2575 scope.go:117] "RemoveContainer" containerID="e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730"
Jul 13 20:39:23.919697 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:39:23.919682    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730\": container with ID starting with e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730 not found: ID does not exist" containerID="e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730"
Jul 13 20:39:23.919754 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.919704    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730"} err="failed to get container status \"e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730\": rpc error: code = NotFound desc = could not find container \"e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730\": container with ID starting with e4e556bfeda30ea556b140c23c1294efb4f2469c2a84a7fc537c24f6aa47d730 not found: ID does not exist"
Jul 13 20:39:23.919754 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.919721    2575 scope.go:117] "RemoveContainer" containerID="a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d"
Jul 13 20:39:23.919981 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:39:23.919961    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d\": container with ID starting with a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d not found: ID does not exist" containerID="a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d"
Jul 13 20:39:23.920054 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:23.920005    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d"} err="failed to get container status \"a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d\": rpc error: code = NotFound desc = could not find container \"a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d\": container with ID starting with a06223ec05c8129ea4bd1002458f631584b62022119243b0e3c31570de6ba09d not found: ID does not exist"
Jul 13 20:39:24.015427 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:24.015395    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:39:24.015427 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:24.015420    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwfmg\" (UniqueName: \"kubernetes.io/projected/16e207be-60cb-4faa-95e9-f6929e0d536b-kube-api-access-hwfmg\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:39:24.015427 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:24.015433    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e207be-60cb-4faa-95e9-f6929e0d536b-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:39:24.226103 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:24.226070    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"]
Jul 13 20:39:24.236937 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:24.236908    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91263-predictor-5f479d8546-d2nfn"]
Jul 13 20:39:25.131059 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:25.131015    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" path="/var/lib/kubelet/pods/16e207be-60cb-4faa-95e9-f6929e0d536b/volumes"
Jul 13 20:39:26.890882 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:26.890853    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:39:26.891381 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:26.891344    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 20:39:27.898866 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:27.898837    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:39:27.899419 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:27.899394    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 20:39:36.892111 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:36.892069    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 20:39:37.900208 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:37.900170    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 20:39:46.891674 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:46.891629    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 20:39:47.899746 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:47.899710    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 20:39:56.891326 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:56.891279    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 20:39:57.900359 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:39:57.900322    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 13 20:40:06.892195 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:06.892111    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:40:07.900233 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:07.900205    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:40:18.972135 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:18.972106    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"]
Jul 13 20:40:18.972553 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:18.972357    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kube-rbac-proxy" containerID="cri-o://3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681" gracePeriod=30
Jul 13 20:40:18.972553 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:18.972334    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container" containerID="cri-o://daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56" gracePeriod=30
Jul 13 20:40:19.010682 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.010653    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"]
Jul 13 20:40:19.011220 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011200    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kube-rbac-proxy"
Jul 13 20:40:19.011220 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011222    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kube-rbac-proxy"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011238    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011247    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011267    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011276    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011297    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kube-rbac-proxy"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011305    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kube-rbac-proxy"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011364    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kube-rbac-proxy"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011377    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be140f10-ffd1-4681-9ca0-4f895724bd3f" containerName="kserve-container"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011388    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kube-rbac-proxy"
Jul 13 20:40:19.011390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.011397    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e207be-60cb-4faa-95e9-f6929e0d536b" containerName="kserve-container"
Jul 13 20:40:19.016190 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.016168    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.020165 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.020139    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e8943-predictor-serving-cert\""
Jul 13 20:40:19.020286 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.020230    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e8943-kube-rbac-proxy-sar-config\""
Jul 13 20:40:19.027623 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.027602    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"]
Jul 13 20:40:19.093170 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.093129    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"]
Jul 13 20:40:19.093641 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.093418    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container" containerID="cri-o://6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f" gracePeriod=30
Jul 13 20:40:19.093641 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.093443    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kube-rbac-proxy" containerID="cri-o://53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c" gracePeriod=30
Jul 13 20:40:19.099237 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.099210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jckpt\" (UniqueName: \"kubernetes.io/projected/e577908b-ec46-4774-83da-4511471d7557-kube-api-access-jckpt\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.100623 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.099253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e577908b-ec46-4774-83da-4511471d7557-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.100623 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.099289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e577908b-ec46-4774-83da-4511471d7557-proxy-tls\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.133371 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.133338    2575 generic.go:356] "Generic (PLEG): container finished" podID="f18dc863-1999-46d0-aa30-37b4c9151890" containerID="3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681" exitCode=2
Jul 13 20:40:19.133718 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.133684    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" event={"ID":"f18dc863-1999-46d0-aa30-37b4c9151890","Type":"ContainerDied","Data":"3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681"}
Jul 13 20:40:19.161961 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.161925    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"]
Jul 13 20:40:19.165805 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.165785    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.169200 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.169180    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e8943-predictor-serving-cert\""
Jul 13 20:40:19.169471 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.169455    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e8943-kube-rbac-proxy-sar-config\""
Jul 13 20:40:19.184393 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.184369    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"]
Jul 13 20:40:19.200346 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.200321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c486b63-5815-4c0d-880b-eaf4ba5c7844-proxy-tls\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.200480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.200358    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgj7c\" (UniqueName: \"kubernetes.io/projected/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-api-access-pgj7c\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.200480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.200392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckpt\" (UniqueName: \"kubernetes.io/projected/e577908b-ec46-4774-83da-4511471d7557-kube-api-access-jckpt\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.200480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.200421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.200480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.200446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e577908b-ec46-4774-83da-4511471d7557-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.200480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.200467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e577908b-ec46-4774-83da-4511471d7557-proxy-tls\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.201194 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.201175    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e577908b-ec46-4774-83da-4511471d7557-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.203428 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.203352    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e577908b-ec46-4774-83da-4511471d7557-proxy-tls\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.212153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.212131    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckpt\" (UniqueName: \"kubernetes.io/projected/e577908b-ec46-4774-83da-4511471d7557-kube-api-access-jckpt\") pod \"success-200-isvc-e8943-predictor-59544c7895-2k9ld\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") " pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.301590 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.301475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c486b63-5815-4c0d-880b-eaf4ba5c7844-proxy-tls\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.301590 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.301544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgj7c\" (UniqueName: \"kubernetes.io/projected/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-api-access-pgj7c\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.301804 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.301592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.302691 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.302660    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.304169 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.304138    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c486b63-5815-4c0d-880b-eaf4ba5c7844-proxy-tls\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.318480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.318459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgj7c\" (UniqueName: \"kubernetes.io/projected/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-api-access-pgj7c\") pod \"error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") " pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.327394 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.327358    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:19.473103 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.473076    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"]
Jul 13 20:40:19.476308 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.476273    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:19.476714 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:40:19.476685    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode577908b_ec46_4774_83da_4511471d7557.slice/crio-4c5f26e37ecf4116c73ea7236f252103441e7baa317dbd3991d0ffb72023cfff WatchSource:0}: Error finding container 4c5f26e37ecf4116c73ea7236f252103441e7baa317dbd3991d0ffb72023cfff: Status 404 returned error can't find the container with id 4c5f26e37ecf4116c73ea7236f252103441e7baa317dbd3991d0ffb72023cfff
Jul 13 20:40:19.640062 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:19.640039    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"]
Jul 13 20:40:19.641514 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:40:19.641486    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c486b63_5815_4c0d_880b_eaf4ba5c7844.slice/crio-b4ef26f962013c09054db33825798763aaad325f2b8e3f5061afb3ae1e3e62c4 WatchSource:0}: Error finding container b4ef26f962013c09054db33825798763aaad325f2b8e3f5061afb3ae1e3e62c4: Status 404 returned error can't find the container with id b4ef26f962013c09054db33825798763aaad325f2b8e3f5061afb3ae1e3e62c4
Jul 13 20:40:20.138880 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.138843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" event={"ID":"e577908b-ec46-4774-83da-4511471d7557","Type":"ContainerStarted","Data":"025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341"}
Jul 13 20:40:20.139350 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.138910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:20.139350 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.138943    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" event={"ID":"e577908b-ec46-4774-83da-4511471d7557","Type":"ContainerStarted","Data":"3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690"}
Jul 13 20:40:20.139350 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.138973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" event={"ID":"e577908b-ec46-4774-83da-4511471d7557","Type":"ContainerStarted","Data":"4c5f26e37ecf4116c73ea7236f252103441e7baa317dbd3991d0ffb72023cfff"}
Jul 13 20:40:20.139350 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.139022    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:20.140539 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.140510    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 20:40:20.140819 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.140786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" event={"ID":"8c486b63-5815-4c0d-880b-eaf4ba5c7844","Type":"ContainerStarted","Data":"4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d"}
Jul 13 20:40:20.140914 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.140848    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:20.140914 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.140872    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:20.140914 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.140898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" event={"ID":"8c486b63-5815-4c0d-880b-eaf4ba5c7844","Type":"ContainerStarted","Data":"3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3"}
Jul 13 20:40:20.141100 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.140917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" event={"ID":"8c486b63-5815-4c0d-880b-eaf4ba5c7844","Type":"ContainerStarted","Data":"b4ef26f962013c09054db33825798763aaad325f2b8e3f5061afb3ae1e3e62c4"}
Jul 13 20:40:20.141936 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.141907    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 20:40:20.142606 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.142577    2575 generic.go:356] "Generic (PLEG): container finished" podID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerID="53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c" exitCode=2
Jul 13 20:40:20.142677 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.142622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" event={"ID":"35b0ae30-4d80-4bb6-b7e9-c071481aba95","Type":"ContainerDied","Data":"53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c"}
Jul 13 20:40:20.164819 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.164778    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podStartSLOduration=2.164765341 podStartE2EDuration="2.164765341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:40:20.162754728 +0000 UTC m=+1265.515363582" watchObservedRunningTime="2026-07-13 20:40:20.164765341 +0000 UTC m=+1265.517374270"
Jul 13 20:40:20.190336 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:20.190287    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podStartSLOduration=1.190271878 podStartE2EDuration="1.190271878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:40:20.189051172 +0000 UTC m=+1265.541660027" watchObservedRunningTime="2026-07-13 20:40:20.190271878 +0000 UTC m=+1265.542880720"
Jul 13 20:40:21.146348 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:21.146310    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 20:40:21.146746 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:21.146406    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 20:40:22.430240 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.430212    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:40:22.463892 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.463868    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:40:22.527076 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527044    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls\") pod \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") "
Jul 13 20:40:22.527076 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f18dc863-1999-46d0-aa30-37b4c9151890-kube-rbac-proxy-sar-config\") pod \"f18dc863-1999-46d0-aa30-37b4c9151890\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") "
Jul 13 20:40:22.527343 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527104    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74x2k\" (UniqueName: \"kubernetes.io/projected/f18dc863-1999-46d0-aa30-37b4c9151890-kube-api-access-74x2k\") pod \"f18dc863-1999-46d0-aa30-37b4c9151890\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") "
Jul 13 20:40:22.527343 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527124    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m79k6\" (UniqueName: \"kubernetes.io/projected/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-api-access-m79k6\") pod \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") "
Jul 13 20:40:22.527343 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527211    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-rbac-proxy-sar-config\") pod \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\" (UID: \"35b0ae30-4d80-4bb6-b7e9-c071481aba95\") "
Jul 13 20:40:22.527343 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527241    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls\") pod \"f18dc863-1999-46d0-aa30-37b4c9151890\" (UID: \"f18dc863-1999-46d0-aa30-37b4c9151890\") "
Jul 13 20:40:22.527608 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527565    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18dc863-1999-46d0-aa30-37b4c9151890-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f18dc863-1999-46d0-aa30-37b4c9151890" (UID: "f18dc863-1999-46d0-aa30-37b4c9151890"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:22.527668 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.527617    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35b0ae30-4d80-4bb6-b7e9-c071481aba95" (UID: "35b0ae30-4d80-4bb6-b7e9-c071481aba95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:22.529388 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.529359    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f18dc863-1999-46d0-aa30-37b4c9151890" (UID: "f18dc863-1999-46d0-aa30-37b4c9151890"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:22.529512 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.529430    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18dc863-1999-46d0-aa30-37b4c9151890-kube-api-access-74x2k" (OuterVolumeSpecName: "kube-api-access-74x2k") pod "f18dc863-1999-46d0-aa30-37b4c9151890" (UID: "f18dc863-1999-46d0-aa30-37b4c9151890"). InnerVolumeSpecName "kube-api-access-74x2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:22.529512 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.529448    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35b0ae30-4d80-4bb6-b7e9-c071481aba95" (UID: "35b0ae30-4d80-4bb6-b7e9-c071481aba95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:22.529512 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.529472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-api-access-m79k6" (OuterVolumeSpecName: "kube-api-access-m79k6") pod "35b0ae30-4d80-4bb6-b7e9-c071481aba95" (UID: "35b0ae30-4d80-4bb6-b7e9-c071481aba95"). InnerVolumeSpecName "kube-api-access-m79k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:22.628673 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.628630    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:22.628673 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.628661    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f18dc863-1999-46d0-aa30-37b4c9151890-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:22.628673 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.628671    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b0ae30-4d80-4bb6-b7e9-c071481aba95-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:22.628673 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.628680    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f18dc863-1999-46d0-aa30-37b4c9151890-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:22.628921 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.628689    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74x2k\" (UniqueName: \"kubernetes.io/projected/f18dc863-1999-46d0-aa30-37b4c9151890-kube-api-access-74x2k\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:22.628921 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:22.628700    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m79k6\" (UniqueName: \"kubernetes.io/projected/35b0ae30-4d80-4bb6-b7e9-c071481aba95-kube-api-access-m79k6\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:23.155711 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.155682    2575 generic.go:356] "Generic (PLEG): container finished" podID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerID="6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f" exitCode=0
Jul 13 20:40:23.155849 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.155739    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"
Jul 13 20:40:23.155849 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.155739    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" event={"ID":"35b0ae30-4d80-4bb6-b7e9-c071481aba95","Type":"ContainerDied","Data":"6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f"}
Jul 13 20:40:23.155951 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.155878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr" event={"ID":"35b0ae30-4d80-4bb6-b7e9-c071481aba95","Type":"ContainerDied","Data":"00645550921afd31e61a11acc7fb0dbc3822f94cc2647f16f070ad59236fe590"}
Jul 13 20:40:23.155951 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.155915    2575 scope.go:117] "RemoveContainer" containerID="53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c"
Jul 13 20:40:23.157339 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.157318    2575 generic.go:356] "Generic (PLEG): container finished" podID="f18dc863-1999-46d0-aa30-37b4c9151890" containerID="daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56" exitCode=0
Jul 13 20:40:23.157486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.157426    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" event={"ID":"f18dc863-1999-46d0-aa30-37b4c9151890","Type":"ContainerDied","Data":"daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56"}
Jul 13 20:40:23.157650 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.157605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc" event={"ID":"f18dc863-1999-46d0-aa30-37b4c9151890","Type":"ContainerDied","Data":"766a783344d48c179e4be9c34ffdcf30786d2736ac62d4c7a6c1e76832077c0c"}
Jul 13 20:40:23.157650 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.157515    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"
Jul 13 20:40:23.164043 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.163966    2575 scope.go:117] "RemoveContainer" containerID="6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f"
Jul 13 20:40:23.171124 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.171102    2575 scope.go:117] "RemoveContainer" containerID="53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c"
Jul 13 20:40:23.171379 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:23.171359    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c\": container with ID starting with 53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c not found: ID does not exist" containerID="53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c"
Jul 13 20:40:23.171424 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.171392    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c"} err="failed to get container status \"53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c\": rpc error: code = NotFound desc = could not find container \"53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c\": container with ID starting with 53202a2f97880ebd1c8d080623cba5b8f0c099424b39cc80e110f8fff6510d6c not found: ID does not exist"
Jul 13 20:40:23.171424 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.171415    2575 scope.go:117] "RemoveContainer" containerID="6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f"
Jul 13 20:40:23.171661 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:23.171642    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f\": container with ID starting with 6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f not found: ID does not exist" containerID="6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f"
Jul 13 20:40:23.171699 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.171666    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f"} err="failed to get container status \"6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f\": rpc error: code = NotFound desc = could not find container \"6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f\": container with ID starting with 6db1012fe7882166c23a1b153ac5676e0aa351e7d4e768c8cb2b178939409c5f not found: ID does not exist"
Jul 13 20:40:23.171699 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.171687    2575 scope.go:117] "RemoveContainer" containerID="3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681"
Jul 13 20:40:23.178810 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.178791    2575 scope.go:117] "RemoveContainer" containerID="daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56"
Jul 13 20:40:23.186519 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.186485    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"]
Jul 13 20:40:23.192460 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.192429    2575 scope.go:117] "RemoveContainer" containerID="3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681"
Jul 13 20:40:23.193974 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:23.193932    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681\": container with ID starting with 3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681 not found: ID does not exist" containerID="3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681"
Jul 13 20:40:23.194193 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.193976    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681"} err="failed to get container status \"3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681\": rpc error: code = NotFound desc = could not find container \"3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681\": container with ID starting with 3bd58cb1406c975faaa5f5ca193c1256f6fc96e2fdb310d377ab019fa159c681 not found: ID does not exist"
Jul 13 20:40:23.194291 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.194197    2575 scope.go:117] "RemoveContainer" containerID="daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56"
Jul 13 20:40:23.194586 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:23.194559    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56\": container with ID starting with daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56 not found: ID does not exist" containerID="daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56"
Jul 13 20:40:23.194655 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.194597    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56"} err="failed to get container status \"daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56\": rpc error: code = NotFound desc = could not find container \"daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56\": container with ID starting with daeb15fe3e6c2566ee70ff072e1f18ceb281c7501d88a0e98e23b80522d09f56 not found: ID does not exist"
Jul 13 20:40:23.195527 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.195511    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43619-predictor-8695456798-vdxlr"]
Jul 13 20:40:23.211040 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.211017    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"]
Jul 13 20:40:23.217099 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:23.217080    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43619-predictor-857f7657f5-b28dc"]
Jul 13 20:40:25.130333 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:25.130298    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" path="/var/lib/kubelet/pods/35b0ae30-4d80-4bb6-b7e9-c071481aba95/volumes"
Jul 13 20:40:25.130699 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:25.130686    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" path="/var/lib/kubelet/pods/f18dc863-1999-46d0-aa30-37b4c9151890/volumes"
Jul 13 20:40:26.150375 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:26.150350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:40:26.150756 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:26.150661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:40:26.150756 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:26.150728    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 20:40:26.151021 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:26.150978    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 20:40:29.578855 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.578821    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"]
Jul 13 20:40:29.579349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.579086    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container" containerID="cri-o://dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3" gracePeriod=30
Jul 13 20:40:29.579349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.579145    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kube-rbac-proxy" containerID="cri-o://08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483" gracePeriod=30
Jul 13 20:40:29.624982 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.624953    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"]
Jul 13 20:40:29.625279 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625267    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container"
Jul 13 20:40:29.625334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625281    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container"
Jul 13 20:40:29.625334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625299    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kube-rbac-proxy"
Jul 13 20:40:29.625334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625305    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kube-rbac-proxy"
Jul 13 20:40:29.625334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625313    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kube-rbac-proxy"
Jul 13 20:40:29.625334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625318    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kube-rbac-proxy"
Jul 13 20:40:29.625334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625325    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container"
Jul 13 20:40:29.625334 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625330    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container"
Jul 13 20:40:29.625549 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625376    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kserve-container"
Jul 13 20:40:29.625549 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625386    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kserve-container"
Jul 13 20:40:29.625549 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625391    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b0ae30-4d80-4bb6-b7e9-c071481aba95" containerName="kube-rbac-proxy"
Jul 13 20:40:29.625549 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.625397    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18dc863-1999-46d0-aa30-37b4c9151890" containerName="kube-rbac-proxy"
Jul 13 20:40:29.629925 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.629894    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.632864 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.632842    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db27e-predictor-serving-cert\""
Jul 13 20:40:29.633057 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.633042    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db27e-kube-rbac-proxy-sar-config\""
Jul 13 20:40:29.640229 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.640207    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"]
Jul 13 20:40:29.686151 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.685716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.686151 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.685818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbpf\" (UniqueName: \"kubernetes.io/projected/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-api-access-vtbpf\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.686151 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.685901    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.696109 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.696077    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"]
Jul 13 20:40:29.696490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.696433    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container" containerID="cri-o://cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0" gracePeriod=30
Jul 13 20:40:29.696490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.696468    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kube-rbac-proxy" containerID="cri-o://d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6" gracePeriod=30
Jul 13 20:40:29.760497 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.760465    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"]
Jul 13 20:40:29.764342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.764320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:29.767629 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.767601    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db27e-predictor-serving-cert\""
Jul 13 20:40:29.767629 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.767610    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db27e-kube-rbac-proxy-sar-config\""
Jul 13 20:40:29.779404 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.779378    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"]
Jul 13 20:40:29.786630 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.786600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.786752 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.786698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.786752 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.786739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbpf\" (UniqueName: \"kubernetes.io/projected/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-api-access-vtbpf\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.787187 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:29.787163    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-db27e-predictor-serving-cert: secret "success-200-isvc-db27e-predictor-serving-cert" not found
Jul 13 20:40:29.787300 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:29.787247    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls podName:a29df3fa-bea3-4590-8ecf-151bc07970c9 nodeName:}" failed. No retries permitted until 2026-07-13 20:40:30.287227037 +0000 UTC m=+1275.639835882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls") pod "success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" (UID: "a29df3fa-bea3-4590-8ecf-151bc07970c9") : secret "success-200-isvc-db27e-predictor-serving-cert" not found
Jul 13 20:40:29.788281 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.788255    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.826139 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.826112    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbpf\" (UniqueName: \"kubernetes.io/projected/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-api-access-vtbpf\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:29.887462 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.887383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnpx\" (UniqueName: \"kubernetes.io/projected/2523fd26-9a63-4029-a430-73d2aa224288-kube-api-access-cjnpx\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:29.887615 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.887573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:29.887669 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.887618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2523fd26-9a63-4029-a430-73d2aa224288-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:29.988562 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.988512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:29.988729 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.988570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2523fd26-9a63-4029-a430-73d2aa224288-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:29.988729 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.988607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnpx\" (UniqueName: \"kubernetes.io/projected/2523fd26-9a63-4029-a430-73d2aa224288-kube-api-access-cjnpx\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:29.988729 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:29.988693    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-db27e-predictor-serving-cert: secret "error-404-isvc-db27e-predictor-serving-cert" not found
Jul 13 20:40:29.988934 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:29.988773    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls podName:2523fd26-9a63-4029-a430-73d2aa224288 nodeName:}" failed. No retries permitted until 2026-07-13 20:40:30.488751071 +0000 UTC m=+1275.841359928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls") pod "error-404-isvc-db27e-predictor-649d5848b4-qx64q" (UID: "2523fd26-9a63-4029-a430-73d2aa224288") : secret "error-404-isvc-db27e-predictor-serving-cert" not found
Jul 13 20:40:29.989537 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:29.989506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2523fd26-9a63-4029-a430-73d2aa224288-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:30.000494 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.000465    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnpx\" (UniqueName: \"kubernetes.io/projected/2523fd26-9a63-4029-a430-73d2aa224288-kube-api-access-cjnpx\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:30.189433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.189334    2575 generic.go:356] "Generic (PLEG): container finished" podID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerID="d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6" exitCode=2
Jul 13 20:40:30.189433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.189413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" event={"ID":"45429bb1-d02d-41de-815c-9aab4b42fb9f","Type":"ContainerDied","Data":"d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6"}
Jul 13 20:40:30.191635 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.191604    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerID="08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483" exitCode=2
Jul 13 20:40:30.191748 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.191701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" event={"ID":"1e9d18b8-9dac-4e40-950d-488bc8b8c49f","Type":"ContainerDied","Data":"08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483"}
Jul 13 20:40:30.291137 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.291100    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:30.294343 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.294314    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls\") pod \"success-200-isvc-db27e-predictor-6c64cf5797-l4fz2\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") " pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:30.493460 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.493418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:30.496086 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.496057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls\") pod \"error-404-isvc-db27e-predictor-649d5848b4-qx64q\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") " pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:30.540910 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.540876    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:30.674711 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.674684    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:30.705584 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.705557    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"]
Jul 13 20:40:30.709976 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:40:30.709940    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29df3fa_bea3_4590_8ecf_151bc07970c9.slice/crio-929cab531abfc7a7b3cae3c4212cc81749a3197b802a7aaad7e3151d2913a10e WatchSource:0}: Error finding container 929cab531abfc7a7b3cae3c4212cc81749a3197b802a7aaad7e3151d2913a10e: Status 404 returned error can't find the container with id 929cab531abfc7a7b3cae3c4212cc81749a3197b802a7aaad7e3151d2913a10e
Jul 13 20:40:30.843104 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:30.843080    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"]
Jul 13 20:40:30.851058 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:40:30.851034    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2523fd26_9a63_4029_a430_73d2aa224288.slice/crio-40eb33827423a7fd78dceb2770e03fc745abab2d9e789c54e45048d562e03dce WatchSource:0}: Error finding container 40eb33827423a7fd78dceb2770e03fc745abab2d9e789c54e45048d562e03dce: Status 404 returned error can't find the container with id 40eb33827423a7fd78dceb2770e03fc745abab2d9e789c54e45048d562e03dce
Jul 13 20:40:31.199195 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.199155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" event={"ID":"2523fd26-9a63-4029-a430-73d2aa224288","Type":"ContainerStarted","Data":"d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3"}
Jul 13 20:40:31.199362 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.199318    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:31.199437 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.199367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" event={"ID":"2523fd26-9a63-4029-a430-73d2aa224288","Type":"ContainerStarted","Data":"6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a"}
Jul 13 20:40:31.199437 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.199401    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" event={"ID":"2523fd26-9a63-4029-a430-73d2aa224288","Type":"ContainerStarted","Data":"40eb33827423a7fd78dceb2770e03fc745abab2d9e789c54e45048d562e03dce"}
Jul 13 20:40:31.199437 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.199432    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:31.200941 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.200913    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 20:40:31.201214 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.201185    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" event={"ID":"a29df3fa-bea3-4590-8ecf-151bc07970c9","Type":"ContainerStarted","Data":"20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13"}
Jul 13 20:40:31.201282 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.201227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" event={"ID":"a29df3fa-bea3-4590-8ecf-151bc07970c9","Type":"ContainerStarted","Data":"4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4"}
Jul 13 20:40:31.201282 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.201254    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:31.201282 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.201272    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:31.201395 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.201288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" event={"ID":"a29df3fa-bea3-4590-8ecf-151bc07970c9","Type":"ContainerStarted","Data":"929cab531abfc7a7b3cae3c4212cc81749a3197b802a7aaad7e3151d2913a10e"}
Jul 13 20:40:31.202178 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.202158    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 20:40:31.225074 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.225037    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podStartSLOduration=2.225024415 podStartE2EDuration="2.225024415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:40:31.224809459 +0000 UTC m=+1276.577418301" watchObservedRunningTime="2026-07-13 20:40:31.225024415 +0000 UTC m=+1276.577633257"
Jul 13 20:40:31.887420 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:31.887376    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 13 20:40:32.205443 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:32.205402    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 20:40:32.205588 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:32.205531    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 20:40:32.896089 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:32.896041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 13 20:40:33.632308 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.632281    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:40:33.660341 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.660246    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podStartSLOduration=4.660226131 podStartE2EDuration="4.660226131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:40:31.252059223 +0000 UTC m=+1276.604668066" watchObservedRunningTime="2026-07-13 20:40:33.660226131 +0000 UTC m=+1279.012834973"
Jul 13 20:40:33.719883 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.719846    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-proxy-tls\") pod \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") "
Jul 13 20:40:33.720075 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.719891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs7z7\" (UniqueName: \"kubernetes.io/projected/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-api-access-gs7z7\") pod \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") "
Jul 13 20:40:33.720075 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.719936    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-rbac-proxy-sar-config\") pod \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\" (UID: \"1e9d18b8-9dac-4e40-950d-488bc8b8c49f\") "
Jul 13 20:40:33.720366 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.720331    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e9d18b8-9dac-4e40-950d-488bc8b8c49f" (UID: "1e9d18b8-9dac-4e40-950d-488bc8b8c49f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:33.722152 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.722122    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e9d18b8-9dac-4e40-950d-488bc8b8c49f" (UID: "1e9d18b8-9dac-4e40-950d-488bc8b8c49f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:33.722252 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.722227    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-api-access-gs7z7" (OuterVolumeSpecName: "kube-api-access-gs7z7") pod "1e9d18b8-9dac-4e40-950d-488bc8b8c49f" (UID: "1e9d18b8-9dac-4e40-950d-488bc8b8c49f"). InnerVolumeSpecName "kube-api-access-gs7z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:33.821448 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.821411    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:33.821448 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.821443    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:33.821448 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:33.821453    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7z7\" (UniqueName: \"kubernetes.io/projected/1e9d18b8-9dac-4e40-950d-488bc8b8c49f-kube-api-access-gs7z7\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:34.215415 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.215377    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerID="dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3" exitCode=0
Jul 13 20:40:34.215868 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.215420    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" event={"ID":"1e9d18b8-9dac-4e40-950d-488bc8b8c49f","Type":"ContainerDied","Data":"dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3"}
Jul 13 20:40:34.215868 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.215426    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"
Jul 13 20:40:34.215868 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.215469    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq" event={"ID":"1e9d18b8-9dac-4e40-950d-488bc8b8c49f","Type":"ContainerDied","Data":"278e251d8102d99b369c1149878e406857b325f3bd79aa97b51902d8860de2b6"}
Jul 13 20:40:34.215868 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.215494    2575 scope.go:117] "RemoveContainer" containerID="08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483"
Jul 13 20:40:34.223942 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.223923    2575 scope.go:117] "RemoveContainer" containerID="dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3"
Jul 13 20:40:34.231281 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.231261    2575 scope.go:117] "RemoveContainer" containerID="08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483"
Jul 13 20:40:34.231512 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:34.231495    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483\": container with ID starting with 08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483 not found: ID does not exist" containerID="08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483"
Jul 13 20:40:34.231553 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.231520    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483"} err="failed to get container status \"08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483\": rpc error: code = NotFound desc = could not find container \"08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483\": container with ID starting with 08f8d535259691d27e60e9ee99e13c267f9d6f5ff61af661a5497fb49d9cc483 not found: ID does not exist"
Jul 13 20:40:34.231553 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.231537    2575 scope.go:117] "RemoveContainer" containerID="dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3"
Jul 13 20:40:34.231781 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:34.231764    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3\": container with ID starting with dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3 not found: ID does not exist" containerID="dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3"
Jul 13 20:40:34.231827 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.231789    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3"} err="failed to get container status \"dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3\": rpc error: code = NotFound desc = could not find container \"dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3\": container with ID starting with dae66476f41b7d776cb8deb566204b64ad31510597934d9dbe9bfd2bb68a39a3 not found: ID does not exist"
Jul 13 20:40:34.240511 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.240491    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"]
Jul 13 20:40:34.244686 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:34.244665    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02ff4-predictor-6d45dbc5c7-j66tq"]
Jul 13 20:40:35.130310 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:35.130278    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" path="/var/lib/kubelet/pods/1e9d18b8-9dac-4e40-950d-488bc8b8c49f/volumes"
Jul 13 20:40:36.150870 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:36.150834    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 20:40:36.151253 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:36.151098    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 20:40:37.210325 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.210294    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:40:37.210787 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.210698    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:40:37.210787 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.210768    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 20:40:37.211157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.211128    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 20:40:37.458383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.458351    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:40:37.549564 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.549479    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls\") pod \"45429bb1-d02d-41de-815c-9aab4b42fb9f\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") "
Jul 13 20:40:37.549564 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.549535    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-rbac-proxy-sar-config\") pod \"45429bb1-d02d-41de-815c-9aab4b42fb9f\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") "
Jul 13 20:40:37.549731 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.549577    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2scr\" (UniqueName: \"kubernetes.io/projected/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-api-access-n2scr\") pod \"45429bb1-d02d-41de-815c-9aab4b42fb9f\" (UID: \"45429bb1-d02d-41de-815c-9aab4b42fb9f\") "
Jul 13 20:40:37.549895 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.549870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45429bb1-d02d-41de-815c-9aab4b42fb9f" (UID: "45429bb1-d02d-41de-815c-9aab4b42fb9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:37.551689 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.551662    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45429bb1-d02d-41de-815c-9aab4b42fb9f" (UID: "45429bb1-d02d-41de-815c-9aab4b42fb9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:37.551807 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.551771    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-api-access-n2scr" (OuterVolumeSpecName: "kube-api-access-n2scr") pod "45429bb1-d02d-41de-815c-9aab4b42fb9f" (UID: "45429bb1-d02d-41de-815c-9aab4b42fb9f"). InnerVolumeSpecName "kube-api-access-n2scr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:37.650672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.650640    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:37.650672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.650667    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2scr\" (UniqueName: \"kubernetes.io/projected/45429bb1-d02d-41de-815c-9aab4b42fb9f-kube-api-access-n2scr\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:37.650672 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:37.650676    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45429bb1-d02d-41de-815c-9aab4b42fb9f-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:40:38.234643 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.234610    2575 generic.go:356] "Generic (PLEG): container finished" podID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerID="cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0" exitCode=0
Jul 13 20:40:38.235153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.234654    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" event={"ID":"45429bb1-d02d-41de-815c-9aab4b42fb9f","Type":"ContainerDied","Data":"cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0"}
Jul 13 20:40:38.235153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.234668    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"
Jul 13 20:40:38.235153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.234702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk" event={"ID":"45429bb1-d02d-41de-815c-9aab4b42fb9f","Type":"ContainerDied","Data":"7632c3c0e1274be522508cc14763ad6325f714140d81b1c6607b2884ea543643"}
Jul 13 20:40:38.235153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.234727    2575 scope.go:117] "RemoveContainer" containerID="d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6"
Jul 13 20:40:38.244379 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.244352    2575 scope.go:117] "RemoveContainer" containerID="cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0"
Jul 13 20:40:38.253477 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.253456    2575 scope.go:117] "RemoveContainer" containerID="d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6"
Jul 13 20:40:38.253758 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:38.253739    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6\": container with ID starting with d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6 not found: ID does not exist" containerID="d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6"
Jul 13 20:40:38.253802 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.253767    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6"} err="failed to get container status \"d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6\": rpc error: code = NotFound desc = could not find container \"d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6\": container with ID starting with d4ce3cda9c88fb8d2d99f9089e17e606ae79843cb69075fd71bc061fbbe4e7a6 not found: ID does not exist"
Jul 13 20:40:38.253802 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.253785    2575 scope.go:117] "RemoveContainer" containerID="cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0"
Jul 13 20:40:38.254061 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:40:38.254041    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0\": container with ID starting with cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0 not found: ID does not exist" containerID="cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0"
Jul 13 20:40:38.254103 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.254069    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0"} err="failed to get container status \"cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0\": rpc error: code = NotFound desc = could not find container \"cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0\": container with ID starting with cf013a10e032a9af05c7d82c5a310d6c0af1a64fc521e7c1acdaad21f351bcb0 not found: ID does not exist"
Jul 13 20:40:38.264352 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.264332    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"]
Jul 13 20:40:38.270039 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:38.270018    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02ff4-predictor-7748bf4588-2k6pk"]
Jul 13 20:40:39.129346 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:39.129308    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" path="/var/lib/kubelet/pods/45429bb1-d02d-41de-815c-9aab4b42fb9f/volumes"
Jul 13 20:40:46.150878 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:46.150843    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 20:40:46.151278 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:46.151074    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 20:40:47.211325 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:47.211285    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 20:40:47.211711 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:47.211286    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 20:40:56.150846 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:56.150804    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 20:40:56.151349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:56.151326    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 20:40:57.211141 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:57.211099    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 20:40:57.211533 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:40:57.211099    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 20:41:06.151297 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:06.151263    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:41:06.151801 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:06.151781    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:41:07.210894 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:07.210834    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 20:41:07.211395 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:07.211118    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 20:41:17.211616 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:17.211589    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:41:17.212047 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:17.211662    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:41:29.252087 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.252051    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"]
Jul 13 20:41:29.252491 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.252309    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container" containerID="cri-o://3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690" gracePeriod=30
Jul 13 20:41:29.252491 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.252368    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kube-rbac-proxy" containerID="cri-o://025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341" gracePeriod=30
Jul 13 20:41:29.324253 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.324204    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"]
Jul 13 20:41:29.324687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.324628    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container" containerID="cri-o://3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3" gracePeriod=30
Jul 13 20:41:29.324815 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.324753    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kube-rbac-proxy" containerID="cri-o://4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d" gracePeriod=30
Jul 13 20:41:29.349164 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349135    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"]
Jul 13 20:41:29.349450 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349437    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kube-rbac-proxy"
Jul 13 20:41:29.349490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349452    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kube-rbac-proxy"
Jul 13 20:41:29.349490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349466    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container"
Jul 13 20:41:29.349490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349472    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container"
Jul 13 20:41:29.349490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349485    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container"
Jul 13 20:41:29.349490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349491    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container"
Jul 13 20:41:29.349631 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349497    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kube-rbac-proxy"
Jul 13 20:41:29.349631 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349503    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kube-rbac-proxy"
Jul 13 20:41:29.349631 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349547    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kube-rbac-proxy"
Jul 13 20:41:29.349631 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349556    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kserve-container"
Jul 13 20:41:29.349631 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349563    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9d18b8-9dac-4e40-950d-488bc8b8c49f" containerName="kserve-container"
Jul 13 20:41:29.349631 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.349569    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45429bb1-d02d-41de-815c-9aab4b42fb9f" containerName="kube-rbac-proxy"
Jul 13 20:41:29.354313 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.354293    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.357860 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.357836    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5859e-predictor-serving-cert\""
Jul 13 20:41:29.358226 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.358201    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5859e-kube-rbac-proxy-sar-config\""
Jul 13 20:41:29.364894 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.364867    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"]
Jul 13 20:41:29.450520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.450470    2575 generic.go:356] "Generic (PLEG): container finished" podID="e577908b-ec46-4774-83da-4511471d7557" containerID="025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341" exitCode=2
Jul 13 20:41:29.450674 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.450546    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" event={"ID":"e577908b-ec46-4774-83da-4511471d7557","Type":"ContainerDied","Data":"025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341"}
Jul 13 20:41:29.453826 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.453799    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b93aa5d-53e3-4993-ae82-3896cc5c6423-proxy-tls\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.453924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.453875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.453978 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.453919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphcx\" (UniqueName: \"kubernetes.io/projected/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-api-access-hphcx\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.455230 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.455196    2575 generic.go:356] "Generic (PLEG): container finished" podID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerID="4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d" exitCode=2
Jul 13 20:41:29.455342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.455226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" event={"ID":"8c486b63-5815-4c0d-880b-eaf4ba5c7844","Type":"ContainerDied","Data":"4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d"}
Jul 13 20:41:29.455342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.455287    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"]
Jul 13 20:41:29.458669 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.458652    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.461715 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.461694    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5859e-predictor-serving-cert\""
Jul 13 20:41:29.462175 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.462159    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5859e-kube-rbac-proxy-sar-config\""
Jul 13 20:41:29.473789 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.473764    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"]
Jul 13 20:41:29.554607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.554522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphcx\" (UniqueName: \"kubernetes.io/projected/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-api-access-hphcx\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.554607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.554564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b93aa5d-53e3-4993-ae82-3896cc5c6423-proxy-tls\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.554607 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.554592    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-proxy-tls\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.554840 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.554662    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.554840 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.554696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.554840 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.554734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvff\" (UniqueName: \"kubernetes.io/projected/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-api-access-gpvff\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.555362 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.555342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.557277 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.557256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b93aa5d-53e3-4993-ae82-3896cc5c6423-proxy-tls\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.566739 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.566712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphcx\" (UniqueName: \"kubernetes.io/projected/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-api-access-hphcx\") pod \"success-200-isvc-5859e-predictor-76c758fc77-sq8sr\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") " pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.656011 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.655932    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.656011 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.656008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvff\" (UniqueName: \"kubernetes.io/projected/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-api-access-gpvff\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.656231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.656058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-proxy-tls\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.657297 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.657256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.658967 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.658941    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-proxy-tls\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.666577 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.666548    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:29.670051 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.670015    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvff\" (UniqueName: \"kubernetes.io/projected/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-api-access-gpvff\") pod \"error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") " pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.768974 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.768920    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:29.813779 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.813284    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"]
Jul 13 20:41:29.816026 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:41:29.815981    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b93aa5d_53e3_4993_ae82_3896cc5c6423.slice/crio-112786474a1ba3ca2d35d4df0db4b6008a84c4f9353675abb594831383574acf WatchSource:0}: Error finding container 112786474a1ba3ca2d35d4df0db4b6008a84c4f9353675abb594831383574acf: Status 404 returned error can't find the container with id 112786474a1ba3ca2d35d4df0db4b6008a84c4f9353675abb594831383574acf
Jul 13 20:41:29.921906 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:29.921876    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"]
Jul 13 20:41:29.925144 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:41:29.925121    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abd4c1c_f51f_41fd_b3c8_f114bfe24b7d.slice/crio-96b44528fde92a21eb729ef99ecf1db68992a6c377269cdd0b97475e768f1676 WatchSource:0}: Error finding container 96b44528fde92a21eb729ef99ecf1db68992a6c377269cdd0b97475e768f1676: Status 404 returned error can't find the container with id 96b44528fde92a21eb729ef99ecf1db68992a6c377269cdd0b97475e768f1676
Jul 13 20:41:30.465816 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.465776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" event={"ID":"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d","Type":"ContainerStarted","Data":"34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67"}
Jul 13 20:41:30.466322 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.465872    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:30.466322 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.465909    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:30.466322 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.465929    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" event={"ID":"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d","Type":"ContainerStarted","Data":"155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856"}
Jul 13 20:41:30.466322 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.465950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" event={"ID":"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d","Type":"ContainerStarted","Data":"96b44528fde92a21eb729ef99ecf1db68992a6c377269cdd0b97475e768f1676"}
Jul 13 20:41:30.467412 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.467385    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 20:41:30.467758 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.467728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" event={"ID":"5b93aa5d-53e3-4993-ae82-3896cc5c6423","Type":"ContainerStarted","Data":"84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1"}
Jul 13 20:41:30.467856 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.467779    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:30.467856 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.467799    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" event={"ID":"5b93aa5d-53e3-4993-ae82-3896cc5c6423","Type":"ContainerStarted","Data":"2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3"}
Jul 13 20:41:30.467856 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.467826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" event={"ID":"5b93aa5d-53e3-4993-ae82-3896cc5c6423","Type":"ContainerStarted","Data":"112786474a1ba3ca2d35d4df0db4b6008a84c4f9353675abb594831383574acf"}
Jul 13 20:41:30.467856 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.467845    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:30.468798 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.468777    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 20:41:30.488839 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.488782    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podStartSLOduration=1.488763184 podStartE2EDuration="1.488763184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:41:30.488573583 +0000 UTC m=+1335.841182426" watchObservedRunningTime="2026-07-13 20:41:30.488763184 +0000 UTC m=+1335.841372032"
Jul 13 20:41:30.509891 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:30.509837    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podStartSLOduration=1.509821861 podStartE2EDuration="1.509821861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:41:30.508766446 +0000 UTC m=+1335.861375308" watchObservedRunningTime="2026-07-13 20:41:30.509821861 +0000 UTC m=+1335.862430702"
Jul 13 20:41:31.146856 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:31.146819    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 20:41:31.147065 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:31.146820    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 13 20:41:31.471902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:31.471860    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 20:41:31.471902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:31.471891    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 20:41:32.707401 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.707377    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:41:32.781248 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.781156    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jckpt\" (UniqueName: \"kubernetes.io/projected/e577908b-ec46-4774-83da-4511471d7557-kube-api-access-jckpt\") pod \"e577908b-ec46-4774-83da-4511471d7557\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") "
Jul 13 20:41:32.781248 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.781210    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e577908b-ec46-4774-83da-4511471d7557-proxy-tls\") pod \"e577908b-ec46-4774-83da-4511471d7557\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") "
Jul 13 20:41:32.781465 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.781271    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e577908b-ec46-4774-83da-4511471d7557-kube-rbac-proxy-sar-config\") pod \"e577908b-ec46-4774-83da-4511471d7557\" (UID: \"e577908b-ec46-4774-83da-4511471d7557\") "
Jul 13 20:41:32.781654 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.781630    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e577908b-ec46-4774-83da-4511471d7557-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e577908b-ec46-4774-83da-4511471d7557" (UID: "e577908b-ec46-4774-83da-4511471d7557"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:41:32.783454 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.783426    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e577908b-ec46-4774-83da-4511471d7557-kube-api-access-jckpt" (OuterVolumeSpecName: "kube-api-access-jckpt") pod "e577908b-ec46-4774-83da-4511471d7557" (UID: "e577908b-ec46-4774-83da-4511471d7557"). InnerVolumeSpecName "kube-api-access-jckpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:41:32.783552 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.783468    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e577908b-ec46-4774-83da-4511471d7557-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e577908b-ec46-4774-83da-4511471d7557" (UID: "e577908b-ec46-4774-83da-4511471d7557"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:41:32.882171 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.882140    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jckpt\" (UniqueName: \"kubernetes.io/projected/e577908b-ec46-4774-83da-4511471d7557-kube-api-access-jckpt\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:41:32.882171 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.882167    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e577908b-ec46-4774-83da-4511471d7557-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:41:32.882171 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:32.882177    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e577908b-ec46-4774-83da-4511471d7557-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:41:33.161488 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.161461    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:41:33.185905 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.185870    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c486b63-5815-4c0d-880b-eaf4ba5c7844-proxy-tls\") pod \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") "
Jul 13 20:41:33.186093 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.185926    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgj7c\" (UniqueName: \"kubernetes.io/projected/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-api-access-pgj7c\") pod \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") "
Jul 13 20:41:33.186093 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.186066    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-rbac-proxy-sar-config\") pod \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\" (UID: \"8c486b63-5815-4c0d-880b-eaf4ba5c7844\") "
Jul 13 20:41:33.186533 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.186448    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c486b63-5815-4c0d-880b-eaf4ba5c7844" (UID: "8c486b63-5815-4c0d-880b-eaf4ba5c7844"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:41:33.188843 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.188801    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c486b63-5815-4c0d-880b-eaf4ba5c7844-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c486b63-5815-4c0d-880b-eaf4ba5c7844" (UID: "8c486b63-5815-4c0d-880b-eaf4ba5c7844"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:41:33.189024 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.188971    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-api-access-pgj7c" (OuterVolumeSpecName: "kube-api-access-pgj7c") pod "8c486b63-5815-4c0d-880b-eaf4ba5c7844" (UID: "8c486b63-5815-4c0d-880b-eaf4ba5c7844"). InnerVolumeSpecName "kube-api-access-pgj7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:41:33.287514 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.287465    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:41:33.287514 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.287495    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c486b63-5815-4c0d-880b-eaf4ba5c7844-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:41:33.287514 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.287505    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgj7c\" (UniqueName: \"kubernetes.io/projected/8c486b63-5815-4c0d-880b-eaf4ba5c7844-kube-api-access-pgj7c\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:41:33.481138 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.481106    2575 generic.go:356] "Generic (PLEG): container finished" podID="e577908b-ec46-4774-83da-4511471d7557" containerID="3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690" exitCode=0
Jul 13 20:41:33.481310 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.481153    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"
Jul 13 20:41:33.481310 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.481195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" event={"ID":"e577908b-ec46-4774-83da-4511471d7557","Type":"ContainerDied","Data":"3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690"}
Jul 13 20:41:33.481310 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.481254    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld" event={"ID":"e577908b-ec46-4774-83da-4511471d7557","Type":"ContainerDied","Data":"4c5f26e37ecf4116c73ea7236f252103441e7baa317dbd3991d0ffb72023cfff"}
Jul 13 20:41:33.481310 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.481292    2575 scope.go:117] "RemoveContainer" containerID="025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341"
Jul 13 20:41:33.482884 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.482856    2575 generic.go:356] "Generic (PLEG): container finished" podID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerID="3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3" exitCode=0
Jul 13 20:41:33.483023 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.482918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" event={"ID":"8c486b63-5815-4c0d-880b-eaf4ba5c7844","Type":"ContainerDied","Data":"3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3"}
Jul 13 20:41:33.483023 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.482956    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"
Jul 13 20:41:33.483023 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.482963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5" event={"ID":"8c486b63-5815-4c0d-880b-eaf4ba5c7844","Type":"ContainerDied","Data":"b4ef26f962013c09054db33825798763aaad325f2b8e3f5061afb3ae1e3e62c4"}
Jul 13 20:41:33.490649 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.490622    2575 scope.go:117] "RemoveContainer" containerID="3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690"
Jul 13 20:41:33.498674 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.498655    2575 scope.go:117] "RemoveContainer" containerID="025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341"
Jul 13 20:41:33.499013 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:41:33.498949    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341\": container with ID starting with 025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341 not found: ID does not exist" containerID="025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341"
Jul 13 20:41:33.499013 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.498984    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341"} err="failed to get container status \"025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341\": rpc error: code = NotFound desc = could not find container \"025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341\": container with ID starting with 025880fddd7899de16076be58a99baf174316e2799a94e5095b8001013e2d341 not found: ID does not exist"
Jul 13 20:41:33.499255 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.499023    2575 scope.go:117] "RemoveContainer" containerID="3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690"
Jul 13 20:41:33.499457 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:41:33.499435    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690\": container with ID starting with 3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690 not found: ID does not exist" containerID="3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690"
Jul 13 20:41:33.499552 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.499464    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690"} err="failed to get container status \"3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690\": rpc error: code = NotFound desc = could not find container \"3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690\": container with ID starting with 3af38fc3b9706433c67bd43a58a516b7fd0df43478b999ce1eed9c6ee181e690 not found: ID does not exist"
Jul 13 20:41:33.499552 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.499481    2575 scope.go:117] "RemoveContainer" containerID="4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d"
Jul 13 20:41:33.501423 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.501400    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"]
Jul 13 20:41:33.506688 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.506665    2575 scope.go:117] "RemoveContainer" containerID="3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3"
Jul 13 20:41:33.509397 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.509377    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e8943-predictor-59544c7895-2k9ld"]
Jul 13 20:41:33.517218 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.517201    2575 scope.go:117] "RemoveContainer" containerID="4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d"
Jul 13 20:41:33.517472 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:41:33.517455    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d\": container with ID starting with 4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d not found: ID does not exist" containerID="4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d"
Jul 13 20:41:33.517522 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.517479    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d"} err="failed to get container status \"4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d\": rpc error: code = NotFound desc = could not find container \"4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d\": container with ID starting with 4e6e2a38ac5f686da031a9b76d673805ac9b79a5be8c9272e2f1d635b218f38d not found: ID does not exist"
Jul 13 20:41:33.517522 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.517493    2575 scope.go:117] "RemoveContainer" containerID="3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3"
Jul 13 20:41:33.517728 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:41:33.517711    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3\": container with ID starting with 3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3 not found: ID does not exist" containerID="3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3"
Jul 13 20:41:33.517775 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.517732    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3"} err="failed to get container status \"3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3\": rpc error: code = NotFound desc = could not find container \"3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3\": container with ID starting with 3723c7b369abe078834502ec837dad74a92e9b907a098cfee298f1dd8e58f6a3 not found: ID does not exist"
Jul 13 20:41:33.532278 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.532255    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"]
Jul 13 20:41:33.544532 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:33.544512    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8943-predictor-6c55cf5cbb-d4pj5"]
Jul 13 20:41:35.129590 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:35.129551    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" path="/var/lib/kubelet/pods/8c486b63-5815-4c0d-880b-eaf4ba5c7844/volumes"
Jul 13 20:41:35.130010 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:35.129948    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e577908b-ec46-4774-83da-4511471d7557" path="/var/lib/kubelet/pods/e577908b-ec46-4774-83da-4511471d7557/volumes"
Jul 13 20:41:36.476866 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:36.476784    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:41:36.477333 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:36.477309    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 20:41:36.478015 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:36.477976    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:41:36.478391 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:36.478372    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 20:41:46.477417 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:46.477372    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 20:41:46.478575 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:46.478552    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 20:41:56.477429 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:56.477391    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 20:41:56.478624 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:41:56.478596    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 20:42:06.477769 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:42:06.477729    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 20:42:06.478378 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:42:06.478354    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 20:42:16.477876 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:42:16.477847    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:42:16.478915 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:42:16.478894    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:44:15.085471 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:44:15.085440    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:44:15.090017 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:44:15.089977    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:49:15.108056 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:15.108024    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:49:15.112620 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:15.112598    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:49:44.490973 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.490940    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"]
Jul 13 20:49:44.491610 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.491292    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" containerID="cri-o://4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4" gracePeriod=30
Jul 13 20:49:44.491610 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.491324    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kube-rbac-proxy" containerID="cri-o://20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13" gracePeriod=30
Jul 13 20:49:44.617575 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.617529    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"]
Jul 13 20:49:44.617968 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.617948    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kube-rbac-proxy"
Jul 13 20:49:44.617968 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.617970    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kube-rbac-proxy"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618010    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618019    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618039    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618048    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618061    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kube-rbac-proxy"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618070    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kube-rbac-proxy"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618144    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kube-rbac-proxy"
Jul 13 20:49:44.618157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618156    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kserve-container"
Jul 13 20:49:44.618520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618170    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e577908b-ec46-4774-83da-4511471d7557" containerName="kube-rbac-proxy"
Jul 13 20:49:44.618520 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.618178    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c486b63-5815-4c0d-880b-eaf4ba5c7844" containerName="kserve-container"
Jul 13 20:49:44.621364 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.621313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.624684 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.624625    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-35019-predictor-serving-cert\""
Jul 13 20:49:44.624795 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.624778    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-35019-kube-rbac-proxy-sar-config\""
Jul 13 20:49:44.657643 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.657610    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"]
Jul 13 20:49:44.662627 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.662597    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"]
Jul 13 20:49:44.662881 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.662851    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" containerID="cri-o://6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a" gracePeriod=30
Jul 13 20:49:44.663020 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.662941    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kube-rbac-proxy" containerID="cri-o://d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3" gracePeriod=30
Jul 13 20:49:44.672513 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.672257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f722fc21-f418-4e36-8e1a-97dbc86f7d27-proxy-tls\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.672513 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.672364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5pr\" (UniqueName: \"kubernetes.io/projected/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-api-access-wb5pr\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.672513 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.672409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.710554 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.710518    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"]
Jul 13 20:49:44.714662 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.714641    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.717201 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.717175    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-35019-kube-rbac-proxy-sar-config\""
Jul 13 20:49:44.717442 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.717412    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-35019-predictor-serving-cert\""
Jul 13 20:49:44.726480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.726455    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"]
Jul 13 20:49:44.773263 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.773179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.773263 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.773247    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6154d545-13b4-434f-8c60-9799415ee4ef-proxy-tls\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.773486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.773302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f722fc21-f418-4e36-8e1a-97dbc86f7d27-proxy-tls\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.773486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.773355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6154d545-13b4-434f-8c60-9799415ee4ef-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.773486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.773406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kq8v\" (UniqueName: \"kubernetes.io/projected/6154d545-13b4-434f-8c60-9799415ee4ef-kube-api-access-6kq8v\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.773486 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.773438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5pr\" (UniqueName: \"kubernetes.io/projected/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-api-access-wb5pr\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.774130 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.774059    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.775974 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.775950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f722fc21-f418-4e36-8e1a-97dbc86f7d27-proxy-tls\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.785408 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.785384    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5pr\" (UniqueName: \"kubernetes.io/projected/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-api-access-wb5pr\") pod \"success-200-isvc-35019-predictor-6647d995d5-4jxvp\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") " pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:44.874332 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.874298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kq8v\" (UniqueName: \"kubernetes.io/projected/6154d545-13b4-434f-8c60-9799415ee4ef-kube-api-access-6kq8v\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.874508 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.874347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6154d545-13b4-434f-8c60-9799415ee4ef-proxy-tls\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.874508 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.874383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6154d545-13b4-434f-8c60-9799415ee4ef-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.874974 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.874943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6154d545-13b4-434f-8c60-9799415ee4ef-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.876962 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.876942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6154d545-13b4-434f-8c60-9799415ee4ef-proxy-tls\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.883472 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.883449    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kq8v\" (UniqueName: \"kubernetes.io/projected/6154d545-13b4-434f-8c60-9799415ee4ef-kube-api-access-6kq8v\") pod \"error-404-isvc-35019-predictor-86b8b845bd-2l9hx\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") " pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:44.934414 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:44.934380    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:45.025620 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.025546    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:45.069451 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.069281    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"]
Jul 13 20:49:45.071576 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:49:45.071548    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf722fc21_f418_4e36_8e1a_97dbc86f7d27.slice/crio-a4b162a3a38f3dc639298fd0eb0570ef55f36c77ba3eca9d3d4daa34de48cfd3 WatchSource:0}: Error finding container a4b162a3a38f3dc639298fd0eb0570ef55f36c77ba3eca9d3d4daa34de48cfd3: Status 404 returned error can't find the container with id a4b162a3a38f3dc639298fd0eb0570ef55f36c77ba3eca9d3d4daa34de48cfd3
Jul 13 20:49:45.170196 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.170174    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"]
Jul 13 20:49:45.179056 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:49:45.179032    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6154d545_13b4_434f_8c60_9799415ee4ef.slice/crio-cbde976e162a3e2ba15e15eb70be7fb8398f36b9a41f3d52ba940957f533b479 WatchSource:0}: Error finding container cbde976e162a3e2ba15e15eb70be7fb8398f36b9a41f3d52ba940957f533b479: Status 404 returned error can't find the container with id cbde976e162a3e2ba15e15eb70be7fb8398f36b9a41f3d52ba940957f533b479
Jul 13 20:49:45.481447 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.481418    2575 generic.go:356] "Generic (PLEG): container finished" podID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerID="20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13" exitCode=2
Jul 13 20:49:45.481632 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.481499    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" event={"ID":"a29df3fa-bea3-4590-8ecf-151bc07970c9","Type":"ContainerDied","Data":"20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13"}
Jul 13 20:49:45.483129 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.483102    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" event={"ID":"6154d545-13b4-434f-8c60-9799415ee4ef","Type":"ContainerStarted","Data":"ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac"}
Jul 13 20:49:45.483236 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.483164    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" event={"ID":"6154d545-13b4-434f-8c60-9799415ee4ef","Type":"ContainerStarted","Data":"e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77"}
Jul 13 20:49:45.483236 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.483200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" event={"ID":"6154d545-13b4-434f-8c60-9799415ee4ef","Type":"ContainerStarted","Data":"cbde976e162a3e2ba15e15eb70be7fb8398f36b9a41f3d52ba940957f533b479"}
Jul 13 20:49:45.483236 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.483235    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:45.483414 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.483396    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:45.485104 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.484917    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 20:49:45.485515 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.485489    2575 generic.go:356] "Generic (PLEG): container finished" podID="2523fd26-9a63-4029-a430-73d2aa224288" containerID="d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3" exitCode=2
Jul 13 20:49:45.485608 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.485587    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" event={"ID":"2523fd26-9a63-4029-a430-73d2aa224288","Type":"ContainerDied","Data":"d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3"}
Jul 13 20:49:45.488965 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.488938    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" event={"ID":"f722fc21-f418-4e36-8e1a-97dbc86f7d27","Type":"ContainerStarted","Data":"6c7550bce4ef6267c1ed50477183e2ce61abdb5c652da533085e5aa61afc75c7"}
Jul 13 20:49:45.489079 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.489030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" event={"ID":"f722fc21-f418-4e36-8e1a-97dbc86f7d27","Type":"ContainerStarted","Data":"f1cb0689c5ca871592d6bc4a6c61a3b8e0d894b161a7af55794b2dc64ee4c17e"}
Jul 13 20:49:45.489079 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.489063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" event={"ID":"f722fc21-f418-4e36-8e1a-97dbc86f7d27","Type":"ContainerStarted","Data":"a4b162a3a38f3dc639298fd0eb0570ef55f36c77ba3eca9d3d4daa34de48cfd3"}
Jul 13 20:49:45.489657 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.489630    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:45.490825 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.489838    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:45.490926 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.490850    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 20:49:45.505771 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.505726    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podStartSLOduration=1.505711993 podStartE2EDuration="1.505711993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:49:45.504130682 +0000 UTC m=+1830.856739525" watchObservedRunningTime="2026-07-13 20:49:45.505711993 +0000 UTC m=+1830.858320835"
Jul 13 20:49:45.523345 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:45.523300    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podStartSLOduration=1.523285591 podStartE2EDuration="1.523285591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:49:45.521790529 +0000 UTC m=+1830.874399382" watchObservedRunningTime="2026-07-13 20:49:45.523285591 +0000 UTC m=+1830.875894434"
Jul 13 20:49:46.494829 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:46.494791    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 20:49:46.495053 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:46.494878    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 20:49:47.205947 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:47.205899    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 13 20:49:47.206346 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:47.205899    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 13 20:49:47.211089 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:47.211063    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 13 20:49:47.211166 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:47.211086    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 20:49:47.498780 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:47.498680    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 20:49:47.887740 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:47.887715    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:49:48.001054 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.001014    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls\") pod \"a29df3fa-bea3-4590-8ecf-151bc07970c9\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") "
Jul 13 20:49:48.001218 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.001121    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtbpf\" (UniqueName: \"kubernetes.io/projected/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-api-access-vtbpf\") pod \"a29df3fa-bea3-4590-8ecf-151bc07970c9\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") "
Jul 13 20:49:48.001218 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.001180    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-rbac-proxy-sar-config\") pod \"a29df3fa-bea3-4590-8ecf-151bc07970c9\" (UID: \"a29df3fa-bea3-4590-8ecf-151bc07970c9\") "
Jul 13 20:49:48.001583 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.001556    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a29df3fa-bea3-4590-8ecf-151bc07970c9" (UID: "a29df3fa-bea3-4590-8ecf-151bc07970c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:49:48.003201 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.003172    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a29df3fa-bea3-4590-8ecf-151bc07970c9" (UID: "a29df3fa-bea3-4590-8ecf-151bc07970c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:49:48.003300 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.003280    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-api-access-vtbpf" (OuterVolumeSpecName: "kube-api-access-vtbpf") pod "a29df3fa-bea3-4590-8ecf-151bc07970c9" (UID: "a29df3fa-bea3-4590-8ecf-151bc07970c9"). InnerVolumeSpecName "kube-api-access-vtbpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:49:48.013665 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.013615    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:49:48.102586 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.102550    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjnpx\" (UniqueName: \"kubernetes.io/projected/2523fd26-9a63-4029-a430-73d2aa224288-kube-api-access-cjnpx\") pod \"2523fd26-9a63-4029-a430-73d2aa224288\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") "
Jul 13 20:49:48.102767 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.102651    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls\") pod \"2523fd26-9a63-4029-a430-73d2aa224288\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") "
Jul 13 20:49:48.102767 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.102699    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2523fd26-9a63-4029-a430-73d2aa224288-kube-rbac-proxy-sar-config\") pod \"2523fd26-9a63-4029-a430-73d2aa224288\" (UID: \"2523fd26-9a63-4029-a430-73d2aa224288\") "
Jul 13 20:49:48.102889 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.102871    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:49:48.102946 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.102894    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a29df3fa-bea3-4590-8ecf-151bc07970c9-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:49:48.102946 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.102910    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtbpf\" (UniqueName: \"kubernetes.io/projected/a29df3fa-bea3-4590-8ecf-151bc07970c9-kube-api-access-vtbpf\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:49:48.103115 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.103091    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2523fd26-9a63-4029-a430-73d2aa224288-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2523fd26-9a63-4029-a430-73d2aa224288" (UID: "2523fd26-9a63-4029-a430-73d2aa224288"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:49:48.104757 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.104736    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2523fd26-9a63-4029-a430-73d2aa224288-kube-api-access-cjnpx" (OuterVolumeSpecName: "kube-api-access-cjnpx") pod "2523fd26-9a63-4029-a430-73d2aa224288" (UID: "2523fd26-9a63-4029-a430-73d2aa224288"). InnerVolumeSpecName "kube-api-access-cjnpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:49:48.104801 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.104748    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2523fd26-9a63-4029-a430-73d2aa224288" (UID: "2523fd26-9a63-4029-a430-73d2aa224288"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:49:48.203547 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.203503    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2523fd26-9a63-4029-a430-73d2aa224288-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:49:48.203547 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.203544    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnpx\" (UniqueName: \"kubernetes.io/projected/2523fd26-9a63-4029-a430-73d2aa224288-kube-api-access-cjnpx\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:49:48.203547 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.203555    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2523fd26-9a63-4029-a430-73d2aa224288-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:49:48.503800 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.503759    2575 generic.go:356] "Generic (PLEG): container finished" podID="2523fd26-9a63-4029-a430-73d2aa224288" containerID="6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a" exitCode=0
Jul 13 20:49:48.504263 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.503820    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"
Jul 13 20:49:48.504263 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.503844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" event={"ID":"2523fd26-9a63-4029-a430-73d2aa224288","Type":"ContainerDied","Data":"6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a"}
Jul 13 20:49:48.504263 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.503898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q" event={"ID":"2523fd26-9a63-4029-a430-73d2aa224288","Type":"ContainerDied","Data":"40eb33827423a7fd78dceb2770e03fc745abab2d9e789c54e45048d562e03dce"}
Jul 13 20:49:48.504263 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.503927    2575 scope.go:117] "RemoveContainer" containerID="d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3"
Jul 13 20:49:48.505613 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.505587    2575 generic.go:356] "Generic (PLEG): container finished" podID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerID="4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4" exitCode=0
Jul 13 20:49:48.505727 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.505648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" event={"ID":"a29df3fa-bea3-4590-8ecf-151bc07970c9","Type":"ContainerDied","Data":"4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4"}
Jul 13 20:49:48.505727 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.505699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2" event={"ID":"a29df3fa-bea3-4590-8ecf-151bc07970c9","Type":"ContainerDied","Data":"929cab531abfc7a7b3cae3c4212cc81749a3197b802a7aaad7e3151d2913a10e"}
Jul 13 20:49:48.505727 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.505698    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"
Jul 13 20:49:48.513505 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.513488    2575 scope.go:117] "RemoveContainer" containerID="6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a"
Jul 13 20:49:48.521135 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.521119    2575 scope.go:117] "RemoveContainer" containerID="d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3"
Jul 13 20:49:48.521406 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:49:48.521387    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3\": container with ID starting with d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3 not found: ID does not exist" containerID="d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3"
Jul 13 20:49:48.521454 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.521415    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3"} err="failed to get container status \"d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3\": rpc error: code = NotFound desc = could not find container \"d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3\": container with ID starting with d56a859750b0337af1767507c24632a83e9a013ba3c89581ebe736ab710690f3 not found: ID does not exist"
Jul 13 20:49:48.521454 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.521431    2575 scope.go:117] "RemoveContainer" containerID="6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a"
Jul 13 20:49:48.521673 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:49:48.521645    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a\": container with ID starting with 6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a not found: ID does not exist" containerID="6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a"
Jul 13 20:49:48.521769 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.521683    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a"} err="failed to get container status \"6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a\": rpc error: code = NotFound desc = could not find container \"6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a\": container with ID starting with 6bf026acdb3703005a9ab5b5298d8588a1c97d09d73b5cbe222180978b39171a not found: ID does not exist"
Jul 13 20:49:48.521769 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.521707    2575 scope.go:117] "RemoveContainer" containerID="20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13"
Jul 13 20:49:48.526533 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.526514    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"]
Jul 13 20:49:48.531765 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.531745    2575 scope.go:117] "RemoveContainer" containerID="4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4"
Jul 13 20:49:48.532279 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.532260    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db27e-predictor-649d5848b4-qx64q"]
Jul 13 20:49:48.539231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.539208    2575 scope.go:117] "RemoveContainer" containerID="20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13"
Jul 13 20:49:48.539675 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:49:48.539627    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13\": container with ID starting with 20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13 not found: ID does not exist" containerID="20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13"
Jul 13 20:49:48.539882 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.539702    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13"} err="failed to get container status \"20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13\": rpc error: code = NotFound desc = could not find container \"20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13\": container with ID starting with 20bb35d69cbdbdcd642cbb752cde0cfee1660ca7f1a56b6c171dd870837c7e13 not found: ID does not exist"
Jul 13 20:49:48.539882 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.539745    2575 scope.go:117] "RemoveContainer" containerID="4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4"
Jul 13 20:49:48.540257 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:49:48.540173    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4\": container with ID starting with 4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4 not found: ID does not exist" containerID="4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4"
Jul 13 20:49:48.540257 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.540208    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4"} err="failed to get container status \"4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4\": rpc error: code = NotFound desc = could not find container \"4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4\": container with ID starting with 4f1710826e94fc33ac8cfe1bdfd46d1040deff6b81ca4365bf3188450382e9f4 not found: ID does not exist"
Jul 13 20:49:48.541458 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.541441    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"]
Jul 13 20:49:48.543869 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:48.543850    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db27e-predictor-6c64cf5797-l4fz2"]
Jul 13 20:49:49.130456 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:49.130425    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2523fd26-9a63-4029-a430-73d2aa224288" path="/var/lib/kubelet/pods/2523fd26-9a63-4029-a430-73d2aa224288/volumes"
Jul 13 20:49:49.130830 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:49.130817    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" path="/var/lib/kubelet/pods/a29df3fa-bea3-4590-8ecf-151bc07970c9/volumes"
Jul 13 20:49:51.499132 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:51.499101    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:49:51.499534 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:51.499509    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 20:49:52.503945 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:52.503917    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:49:52.504361 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:49:52.504339    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 20:50:01.500490 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:01.500446    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 20:50:02.504733 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:02.504693    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 20:50:11.499688 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:11.499647    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 20:50:12.504853 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:12.504811    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 20:50:21.499507 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:21.499463    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 13 20:50:22.505353 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:22.505316    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 20:50:31.500832 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:31.500801    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:50:32.505143 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:32.505110    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:50:44.085319 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.085237    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"]
Jul 13 20:50:44.085837 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.085508    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" containerID="cri-o://2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3" gracePeriod=30
Jul 13 20:50:44.085837 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.085552    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kube-rbac-proxy" containerID="cri-o://84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1" gracePeriod=30
Jul 13 20:50:44.128242 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.128210    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"]
Jul 13 20:50:44.128480 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.128454    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" containerID="cri-o://155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856" gracePeriod=30
Jul 13 20:50:44.128609 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.128585    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kube-rbac-proxy" containerID="cri-o://34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67" gracePeriod=30
Jul 13 20:50:44.184054 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184019    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"]
Jul 13 20:50:44.184493 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184478    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container"
Jul 13 20:50:44.184539 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184496    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container"
Jul 13 20:50:44.184539 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184514    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kube-rbac-proxy"
Jul 13 20:50:44.184539 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184522    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kube-rbac-proxy"
Jul 13 20:50:44.184539 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184536    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container"
Jul 13 20:50:44.184659 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184544    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container"
Jul 13 20:50:44.184659 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184557    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kube-rbac-proxy"
Jul 13 20:50:44.184659 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184565    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kube-rbac-proxy"
Jul 13 20:50:44.184659 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184635    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kserve-container"
Jul 13 20:50:44.184659 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184647    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kserve-container"
Jul 13 20:50:44.184659 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184656    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2523fd26-9a63-4029-a430-73d2aa224288" containerName="kube-rbac-proxy"
Jul 13 20:50:44.184825 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.184667    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29df3fa-bea3-4590-8ecf-151bc07970c9" containerName="kube-rbac-proxy"
Jul 13 20:50:44.188182 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.188160    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.191181 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.190731    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-575ad-kube-rbac-proxy-sar-config\""
Jul 13 20:50:44.191181 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.191061    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-575ad-predictor-serving-cert\""
Jul 13 20:50:44.196457 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.196361    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"]
Jul 13 20:50:44.245092 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.245062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.245267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.245135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65s6m\" (UniqueName: \"kubernetes.io/projected/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-api-access-65s6m\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.245267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.245186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.278897 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.278856    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"]
Jul 13 20:50:44.282329 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.282308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.284936 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.284916    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-575ad-predictor-serving-cert\""
Jul 13 20:50:44.285075 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.284937    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-575ad-kube-rbac-proxy-sar-config\""
Jul 13 20:50:44.293755 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.293732    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"]
Jul 13 20:50:44.346485 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.346396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.346485 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.346460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.346692 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.346492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc85ef38-2865-43c7-a334-96fc8736d535-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.346692 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.346539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.346692 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.346573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2snm\" (UniqueName: \"kubernetes.io/projected/bc85ef38-2865-43c7-a334-96fc8736d535-kube-api-access-b2snm\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.346692 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.346636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65s6m\" (UniqueName: \"kubernetes.io/projected/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-api-access-65s6m\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.346692 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:44.346648    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-575ad-predictor-serving-cert: secret "success-200-isvc-575ad-predictor-serving-cert" not found
Jul 13 20:50:44.346957 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:44.346728    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls podName:1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d nodeName:}" failed. No retries permitted until 2026-07-13 20:50:44.846707007 +0000 UTC m=+1890.199315836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls") pod "success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" (UID: "1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d") : secret "success-200-isvc-575ad-predictor-serving-cert" not found
Jul 13 20:50:44.347265 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.347215    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.356291 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.356267    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65s6m\" (UniqueName: \"kubernetes.io/projected/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-api-access-65s6m\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.447049 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.447015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.447204 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.447069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc85ef38-2865-43c7-a334-96fc8736d535-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.447204 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:44.447092    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-575ad-predictor-serving-cert: secret "error-404-isvc-575ad-predictor-serving-cert" not found
Jul 13 20:50:44.447204 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.447143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2snm\" (UniqueName: \"kubernetes.io/projected/bc85ef38-2865-43c7-a334-96fc8736d535-kube-api-access-b2snm\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.447204 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:44.447156    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls podName:bc85ef38-2865-43c7-a334-96fc8736d535 nodeName:}" failed. No retries permitted until 2026-07-13 20:50:44.947136569 +0000 UTC m=+1890.299745392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls") pod "error-404-isvc-575ad-predictor-7d77b767d4-xbql6" (UID: "bc85ef38-2865-43c7-a334-96fc8736d535") : secret "error-404-isvc-575ad-predictor-serving-cert" not found
Jul 13 20:50:44.447725 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.447705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc85ef38-2865-43c7-a334-96fc8736d535-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.455898 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.455869    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2snm\" (UniqueName: \"kubernetes.io/projected/bc85ef38-2865-43c7-a334-96fc8736d535-kube-api-access-b2snm\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.744207 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.744172    2575 generic.go:356] "Generic (PLEG): container finished" podID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerID="34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67" exitCode=2
Jul 13 20:50:44.744396 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.744241    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" event={"ID":"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d","Type":"ContainerDied","Data":"34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67"}
Jul 13 20:50:44.745984 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.745961    2575 generic.go:356] "Generic (PLEG): container finished" podID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerID="84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1" exitCode=2
Jul 13 20:50:44.746107 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.746014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" event={"ID":"5b93aa5d-53e3-4993-ae82-3896cc5c6423","Type":"ContainerDied","Data":"84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1"}
Jul 13 20:50:44.850446 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.850408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.853050 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.853026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls\") pod \"success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") " pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:44.951697 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.951665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:44.954128 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:44.954099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls\") pod \"error-404-isvc-575ad-predictor-7d77b767d4-xbql6\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") " pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:45.102825 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.102746    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:45.193555 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.193513    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:45.241679 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.241597    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"]
Jul 13 20:50:45.245053 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:50:45.245029    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf0ceca_cfb7_49b1_9f7c_4e9bff37a15d.slice/crio-7a68ff3664cb84328580eef661ad4b18dcdb715c16ee1c7d94eb3048db5c6d0a WatchSource:0}: Error finding container 7a68ff3664cb84328580eef661ad4b18dcdb715c16ee1c7d94eb3048db5c6d0a: Status 404 returned error can't find the container with id 7a68ff3664cb84328580eef661ad4b18dcdb715c16ee1c7d94eb3048db5c6d0a
Jul 13 20:50:45.338513 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.338471    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"]
Jul 13 20:50:45.340516 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:50:45.340494    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc85ef38_2865_43c7_a334_96fc8736d535.slice/crio-0e4b510808615739599cae39c8177fb4817e366fc9813302a7a176b89d936e4a WatchSource:0}: Error finding container 0e4b510808615739599cae39c8177fb4817e366fc9813302a7a176b89d936e4a: Status 404 returned error can't find the container with id 0e4b510808615739599cae39c8177fb4817e366fc9813302a7a176b89d936e4a
Jul 13 20:50:45.751921 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.751888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" event={"ID":"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d","Type":"ContainerStarted","Data":"8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4"}
Jul 13 20:50:45.752133 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.752080    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:45.752206 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.752133    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" event={"ID":"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d","Type":"ContainerStarted","Data":"42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e"}
Jul 13 20:50:45.752206 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.752190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" event={"ID":"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d","Type":"ContainerStarted","Data":"7a68ff3664cb84328580eef661ad4b18dcdb715c16ee1c7d94eb3048db5c6d0a"}
Jul 13 20:50:45.752301 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.752224    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:45.753651 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.753622    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 20:50:45.753924 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.753896    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" event={"ID":"bc85ef38-2865-43c7-a334-96fc8736d535","Type":"ContainerStarted","Data":"3152710d5b6bbd4e4a982b5461199447fab0dd4dedc0f12dbba76a74594cfed1"}
Jul 13 20:50:45.754036 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.753951    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" event={"ID":"bc85ef38-2865-43c7-a334-96fc8736d535","Type":"ContainerStarted","Data":"deab5066495b770f2d857269d6a7790008ffd0499af249fdf7b66254e9f4492a"}
Jul 13 20:50:45.754036 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.753984    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" event={"ID":"bc85ef38-2865-43c7-a334-96fc8736d535","Type":"ContainerStarted","Data":"0e4b510808615739599cae39c8177fb4817e366fc9813302a7a176b89d936e4a"}
Jul 13 20:50:45.754118 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.754044    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:45.773082 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.773032    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podStartSLOduration=1.7730187210000001 podStartE2EDuration="1.773018721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:50:45.77071704 +0000 UTC m=+1891.123325895" watchObservedRunningTime="2026-07-13 20:50:45.773018721 +0000 UTC m=+1891.125627563"
Jul 13 20:50:45.788374 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:45.788327    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podStartSLOduration=1.788308179 podStartE2EDuration="1.788308179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:50:45.78694909 +0000 UTC m=+1891.139557940" watchObservedRunningTime="2026-07-13 20:50:45.788308179 +0000 UTC m=+1891.140917031"
Jul 13 20:50:46.472219 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:46.472181    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 13 20:50:46.472575 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:46.472188    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 13 20:50:46.477536 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:46.477511    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 20:50:46.478714 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:46.478696    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 13 20:50:46.757854 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:46.757755    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:46.757854 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:46.757772    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 20:50:46.759064 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:46.759041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 20:50:47.582118 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.582091    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:50:47.676177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.676094    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-rbac-proxy-sar-config\") pod \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") "
Jul 13 20:50:47.676177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.676139    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpvff\" (UniqueName: \"kubernetes.io/projected/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-api-access-gpvff\") pod \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") "
Jul 13 20:50:47.676177 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.676170    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-proxy-tls\") pod \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\" (UID: \"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d\") "
Jul 13 20:50:47.676504 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.676480    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" (UID: "2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:50:47.678351 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.678328    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" (UID: "2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:50:47.678423 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.678392    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-api-access-gpvff" (OuterVolumeSpecName: "kube-api-access-gpvff") pod "2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" (UID: "2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d"). InnerVolumeSpecName "kube-api-access-gpvff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:50:47.763182 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.763149    2575 generic.go:356] "Generic (PLEG): container finished" podID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerID="155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856" exitCode=0
Jul 13 20:50:47.763364 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.763200    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"
Jul 13 20:50:47.763364 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.763234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" event={"ID":"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d","Type":"ContainerDied","Data":"155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856"}
Jul 13 20:50:47.763364 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.763292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d" event={"ID":"2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d","Type":"ContainerDied","Data":"96b44528fde92a21eb729ef99ecf1db68992a6c377269cdd0b97475e768f1676"}
Jul 13 20:50:47.763364 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.763318    2575 scope.go:117] "RemoveContainer" containerID="34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67"
Jul 13 20:50:47.763782 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.763759    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 20:50:47.772229 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.772212    2575 scope.go:117] "RemoveContainer" containerID="155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856"
Jul 13 20:50:47.777660 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.777641    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:47.777763 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.777662    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:47.777763 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.777676    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvff\" (UniqueName: \"kubernetes.io/projected/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d-kube-api-access-gpvff\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:47.779463 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.779446    2575 scope.go:117] "RemoveContainer" containerID="34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67"
Jul 13 20:50:47.779695 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:47.779677    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67\": container with ID starting with 34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67 not found: ID does not exist" containerID="34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67"
Jul 13 20:50:47.779737 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.779704    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67"} err="failed to get container status \"34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67\": rpc error: code = NotFound desc = could not find container \"34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67\": container with ID starting with 34fdaa052e6df7f9d9c7b38d5a8385d525470821ed43bbd455c70a000079de67 not found: ID does not exist"
Jul 13 20:50:47.779737 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.779722    2575 scope.go:117] "RemoveContainer" containerID="155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856"
Jul 13 20:50:47.782389 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:47.782362    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856\": container with ID starting with 155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856 not found: ID does not exist" containerID="155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856"
Jul 13 20:50:47.782491 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.782400    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856"} err="failed to get container status \"155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856\": rpc error: code = NotFound desc = could not find container \"155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856\": container with ID starting with 155c581e48545503e988c1533bf859a36a9b5382a7440a13b9c3c54f610db856 not found: ID does not exist"
Jul 13 20:50:47.784065 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.784041    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"]
Jul 13 20:50:47.789636 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:47.789612    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5859e-predictor-55f99fd8b8-5mg6d"]
Jul 13 20:50:48.029542 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.029517    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:50:48.079935 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.079904    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-rbac-proxy-sar-config\") pod \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") "
Jul 13 20:50:48.080128 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.079947    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hphcx\" (UniqueName: \"kubernetes.io/projected/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-api-access-hphcx\") pod \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") "
Jul 13 20:50:48.080128 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.080004    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b93aa5d-53e3-4993-ae82-3896cc5c6423-proxy-tls\") pod \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\" (UID: \"5b93aa5d-53e3-4993-ae82-3896cc5c6423\") "
Jul 13 20:50:48.080380 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.080355    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b93aa5d-53e3-4993-ae82-3896cc5c6423" (UID: "5b93aa5d-53e3-4993-ae82-3896cc5c6423"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:50:48.082246 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.082225    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b93aa5d-53e3-4993-ae82-3896cc5c6423-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b93aa5d-53e3-4993-ae82-3896cc5c6423" (UID: "5b93aa5d-53e3-4993-ae82-3896cc5c6423"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:50:48.082307 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.082291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-api-access-hphcx" (OuterVolumeSpecName: "kube-api-access-hphcx") pod "5b93aa5d-53e3-4993-ae82-3896cc5c6423" (UID: "5b93aa5d-53e3-4993-ae82-3896cc5c6423"). InnerVolumeSpecName "kube-api-access-hphcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:50:48.181033 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.180977    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:48.181033 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.181024    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphcx\" (UniqueName: \"kubernetes.io/projected/5b93aa5d-53e3-4993-ae82-3896cc5c6423-kube-api-access-hphcx\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:48.181033 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.181035    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b93aa5d-53e3-4993-ae82-3896cc5c6423-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:48.770297 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.770265    2575 generic.go:356] "Generic (PLEG): container finished" podID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerID="2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3" exitCode=0
Jul 13 20:50:48.770687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.770310    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"
Jul 13 20:50:48.770687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.770349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" event={"ID":"5b93aa5d-53e3-4993-ae82-3896cc5c6423","Type":"ContainerDied","Data":"2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3"}
Jul 13 20:50:48.770687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.770405    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr" event={"ID":"5b93aa5d-53e3-4993-ae82-3896cc5c6423","Type":"ContainerDied","Data":"112786474a1ba3ca2d35d4df0db4b6008a84c4f9353675abb594831383574acf"}
Jul 13 20:50:48.770687 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.770433    2575 scope.go:117] "RemoveContainer" containerID="84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1"
Jul 13 20:50:48.779152 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.779135    2575 scope.go:117] "RemoveContainer" containerID="2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3"
Jul 13 20:50:48.787597 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.787571    2575 scope.go:117] "RemoveContainer" containerID="84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1"
Jul 13 20:50:48.787911 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:48.787893    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1\": container with ID starting with 84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1 not found: ID does not exist" containerID="84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1"
Jul 13 20:50:48.787980 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.787920    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1"} err="failed to get container status \"84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1\": rpc error: code = NotFound desc = could not find container \"84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1\": container with ID starting with 84b5ccf1e2c9819f4956b87bd8d84c7bc37f2d567f51953721a8cde3020e71c1 not found: ID does not exist"
Jul 13 20:50:48.787980 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.787937    2575 scope.go:117] "RemoveContainer" containerID="2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3"
Jul 13 20:50:48.788272 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:48.788255    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3\": container with ID starting with 2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3 not found: ID does not exist" containerID="2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3"
Jul 13 20:50:48.788311 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.788277    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3"} err="failed to get container status \"2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3\": rpc error: code = NotFound desc = could not find container \"2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3\": container with ID starting with 2cfa77f5446d3b50ef335af3908e2963b1cadbd25857eaa5200a3d4e8ac1b6b3 not found: ID does not exist"
Jul 13 20:50:48.792126 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.792103    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"]
Jul 13 20:50:48.795375 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:48.795357    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5859e-predictor-76c758fc77-sq8sr"]
Jul 13 20:50:49.131431 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:49.131352    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" path="/var/lib/kubelet/pods/2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d/volumes"
Jul 13 20:50:49.131764 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:49.131751    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" path="/var/lib/kubelet/pods/5b93aa5d-53e3-4993-ae82-3896cc5c6423/volumes"
Jul 13 20:50:51.762743 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:51.762712    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:50:51.763214 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:51.763184    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 20:50:52.768616 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:52.768589    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:50:52.769080 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:52.769043    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 20:50:54.779307 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.779276    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"]
Jul 13 20:50:54.779728 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.779501    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container" containerID="cri-o://f1cb0689c5ca871592d6bc4a6c61a3b8e0d894b161a7af55794b2dc64ee4c17e" gracePeriod=30
Jul 13 20:50:54.779728 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.779546    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kube-rbac-proxy" containerID="cri-o://6c7550bce4ef6267c1ed50477183e2ce61abdb5c652da533085e5aa61afc75c7" gracePeriod=30
Jul 13 20:50:54.829390 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.829347    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"]
Jul 13 20:50:54.829657 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.829624    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container" containerID="cri-o://e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77" gracePeriod=30
Jul 13 20:50:54.829801 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.829668    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kube-rbac-proxy" containerID="cri-o://ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac" gracePeriod=30
Jul 13 20:50:54.853609 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853583    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"]
Jul 13 20:50:54.853894 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853883    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kube-rbac-proxy"
Jul 13 20:50:54.853934 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853896    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kube-rbac-proxy"
Jul 13 20:50:54.853934 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853905    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container"
Jul 13 20:50:54.853934 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853910    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container"
Jul 13 20:50:54.853934 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853921    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container"
Jul 13 20:50:54.853934 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853926    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container"
Jul 13 20:50:54.853934 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853935    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kube-rbac-proxy"
Jul 13 20:50:54.854342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.853941    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kube-rbac-proxy"
Jul 13 20:50:54.854342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.854019    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kube-rbac-proxy"
Jul 13 20:50:54.854342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.854030    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kserve-container"
Jul 13 20:50:54.854342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.854037    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b93aa5d-53e3-4993-ae82-3896cc5c6423" containerName="kserve-container"
Jul 13 20:50:54.854342 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.854044    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abd4c1c-f51f-41fd-b3c8-f114bfe24b7d" containerName="kube-rbac-proxy"
Jul 13 20:50:54.858084 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.858057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:54.860666 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.860638    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-492f6-predictor-serving-cert\""
Jul 13 20:50:54.860786 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.860698    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-492f6-kube-rbac-proxy-sar-config\""
Jul 13 20:50:54.867717 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.867690    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"]
Jul 13 20:50:54.932536 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.932504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:54.932679 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.932568    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glgj\" (UniqueName: \"kubernetes.io/projected/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-api-access-9glgj\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:54.932679 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.932597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5fc5723-4d05-4cea-ba54-22cea2393bec-proxy-tls\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:54.949499 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.949474    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"]
Jul 13 20:50:54.952788 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.952769    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:54.956902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.956883    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-492f6-kube-rbac-proxy-sar-config\""
Jul 13 20:50:54.957039 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.956934    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-492f6-predictor-serving-cert\""
Jul 13 20:50:54.964656 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:54.964632    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"]
Jul 13 20:50:55.033729 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.033657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppxx\" (UniqueName: \"kubernetes.io/projected/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-api-access-9ppxx\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.033729 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.033704    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.033902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.033729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.033902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.033778    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63603a40-71b1-470e-bc4c-424e7c0f418c-proxy-tls\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.033902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.033824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glgj\" (UniqueName: \"kubernetes.io/projected/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-api-access-9glgj\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.033902 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.033852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5fc5723-4d05-4cea-ba54-22cea2393bec-proxy-tls\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.034414 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.034396    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.036697 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.036672    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5fc5723-4d05-4cea-ba54-22cea2393bec-proxy-tls\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.041551 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.041523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glgj\" (UniqueName: \"kubernetes.io/projected/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-api-access-9glgj\") pod \"success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") " pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.134679 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.134648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppxx\" (UniqueName: \"kubernetes.io/projected/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-api-access-9ppxx\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.134860 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.134697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.134860 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.134822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63603a40-71b1-470e-bc4c-424e7c0f418c-proxy-tls\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.135384 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.135367    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.137431 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.137409    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63603a40-71b1-470e-bc4c-424e7c0f418c-proxy-tls\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.144199 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.144176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppxx\" (UniqueName: \"kubernetes.io/projected/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-api-access-9ppxx\") pod \"error-404-isvc-492f6-predictor-598d856bcd-kxgf8\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") " pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.170086 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.170062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.264383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.264352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.308898 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.308766    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"]
Jul 13 20:50:55.315778 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:50:55.314104    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fc5723_4d05_4cea_ba54_22cea2393bec.slice/crio-bedcc9d08ebc39d01d5497c503a57073d1ffdaa8d1a0bb288844ee0516d971fa WatchSource:0}: Error finding container bedcc9d08ebc39d01d5497c503a57073d1ffdaa8d1a0bb288844ee0516d971fa: Status 404 returned error can't find the container with id bedcc9d08ebc39d01d5497c503a57073d1ffdaa8d1a0bb288844ee0516d971fa
Jul 13 20:50:55.410459 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.410354    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"]
Jul 13 20:50:55.415164 ip-10-0-128-233 kubenswrapper[2575]: W0713 20:50:55.415142    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63603a40_71b1_470e_bc4c_424e7c0f418c.slice/crio-864a745253aae85c2c801d0a99df3ae614cdc0f5f7ba8334a9653e9090108a5b WatchSource:0}: Error finding container 864a745253aae85c2c801d0a99df3ae614cdc0f5f7ba8334a9653e9090108a5b: Status 404 returned error can't find the container with id 864a745253aae85c2c801d0a99df3ae614cdc0f5f7ba8334a9653e9090108a5b
Jul 13 20:50:55.806515 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.806480    2575 generic.go:356] "Generic (PLEG): container finished" podID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerID="6c7550bce4ef6267c1ed50477183e2ce61abdb5c652da533085e5aa61afc75c7" exitCode=2
Jul 13 20:50:55.806892 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.806550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" event={"ID":"f722fc21-f418-4e36-8e1a-97dbc86f7d27","Type":"ContainerDied","Data":"6c7550bce4ef6267c1ed50477183e2ce61abdb5c652da533085e5aa61afc75c7"}
Jul 13 20:50:55.808231 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.808211    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" event={"ID":"63603a40-71b1-470e-bc4c-424e7c0f418c","Type":"ContainerStarted","Data":"7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1"}
Jul 13 20:50:55.808349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.808254    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" event={"ID":"63603a40-71b1-470e-bc4c-424e7c0f418c","Type":"ContainerStarted","Data":"9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787"}
Jul 13 20:50:55.808349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.808273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" event={"ID":"63603a40-71b1-470e-bc4c-424e7c0f418c","Type":"ContainerStarted","Data":"864a745253aae85c2c801d0a99df3ae614cdc0f5f7ba8334a9653e9090108a5b"}
Jul 13 20:50:55.808349 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.808301    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.808518 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.808419    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:50:55.809791 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.809770    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 20:50:55.810107 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.810087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" event={"ID":"f5fc5723-4d05-4cea-ba54-22cea2393bec","Type":"ContainerStarted","Data":"1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d"}
Jul 13 20:50:55.810193 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.810135    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.810193 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.810165    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:50:55.810193 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.810184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" event={"ID":"f5fc5723-4d05-4cea-ba54-22cea2393bec","Type":"ContainerStarted","Data":"100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f"}
Jul 13 20:50:55.810336 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.810201    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" event={"ID":"f5fc5723-4d05-4cea-ba54-22cea2393bec","Type":"ContainerStarted","Data":"bedcc9d08ebc39d01d5497c503a57073d1ffdaa8d1a0bb288844ee0516d971fa"}
Jul 13 20:50:55.811148 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.811127    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 20:50:55.811895 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.811877    2575 generic.go:356] "Generic (PLEG): container finished" podID="6154d545-13b4-434f-8c60-9799415ee4ef" containerID="ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac" exitCode=2
Jul 13 20:50:55.811966 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.811916    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" event={"ID":"6154d545-13b4-434f-8c60-9799415ee4ef","Type":"ContainerDied","Data":"ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac"}
Jul 13 20:50:55.827283 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.827235    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podStartSLOduration=1.8272226040000001 podStartE2EDuration="1.827222604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:50:55.825798655 +0000 UTC m=+1901.178407499" watchObservedRunningTime="2026-07-13 20:50:55.827222604 +0000 UTC m=+1901.179831446"
Jul 13 20:50:55.843929 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:55.843892    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podStartSLOduration=1.8438791719999998 podStartE2EDuration="1.843879172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:50:55.84323645 +0000 UTC m=+1901.195845285" watchObservedRunningTime="2026-07-13 20:50:55.843879172 +0000 UTC m=+1901.196488015"
Jul 13 20:50:56.495624 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:56.495584    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 13 20:50:56.816417 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:56.816326    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 20:50:56.816761 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:56.816431    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 20:50:57.499380 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:57.499341    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 13 20:50:58.591706 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.591670    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:50:58.664313 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.664217    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6154d545-13b4-434f-8c60-9799415ee4ef-kube-rbac-proxy-sar-config\") pod \"6154d545-13b4-434f-8c60-9799415ee4ef\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") "
Jul 13 20:50:58.664313 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.664308    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6154d545-13b4-434f-8c60-9799415ee4ef-proxy-tls\") pod \"6154d545-13b4-434f-8c60-9799415ee4ef\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") "
Jul 13 20:50:58.664527 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.664334    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kq8v\" (UniqueName: \"kubernetes.io/projected/6154d545-13b4-434f-8c60-9799415ee4ef-kube-api-access-6kq8v\") pod \"6154d545-13b4-434f-8c60-9799415ee4ef\" (UID: \"6154d545-13b4-434f-8c60-9799415ee4ef\") "
Jul 13 20:50:58.664629 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.664600    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6154d545-13b4-434f-8c60-9799415ee4ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6154d545-13b4-434f-8c60-9799415ee4ef" (UID: "6154d545-13b4-434f-8c60-9799415ee4ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:50:58.666512 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.666487    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6154d545-13b4-434f-8c60-9799415ee4ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6154d545-13b4-434f-8c60-9799415ee4ef" (UID: "6154d545-13b4-434f-8c60-9799415ee4ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:50:58.666568 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.666535    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6154d545-13b4-434f-8c60-9799415ee4ef-kube-api-access-6kq8v" (OuterVolumeSpecName: "kube-api-access-6kq8v") pod "6154d545-13b4-434f-8c60-9799415ee4ef" (UID: "6154d545-13b4-434f-8c60-9799415ee4ef"). InnerVolumeSpecName "kube-api-access-6kq8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:50:58.765161 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.765126    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6154d545-13b4-434f-8c60-9799415ee4ef-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:58.765161 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.765155    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kq8v\" (UniqueName: \"kubernetes.io/projected/6154d545-13b4-434f-8c60-9799415ee4ef-kube-api-access-6kq8v\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:58.765161 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.765165    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6154d545-13b4-434f-8c60-9799415ee4ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:58.827150 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.827116    2575 generic.go:356] "Generic (PLEG): container finished" podID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerID="f1cb0689c5ca871592d6bc4a6c61a3b8e0d894b161a7af55794b2dc64ee4c17e" exitCode=0
Jul 13 20:50:58.827333 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.827183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" event={"ID":"f722fc21-f418-4e36-8e1a-97dbc86f7d27","Type":"ContainerDied","Data":"f1cb0689c5ca871592d6bc4a6c61a3b8e0d894b161a7af55794b2dc64ee4c17e"}
Jul 13 20:50:58.828844 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.828819    2575 generic.go:356] "Generic (PLEG): container finished" podID="6154d545-13b4-434f-8c60-9799415ee4ef" containerID="e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77" exitCode=0
Jul 13 20:50:58.829267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.828850    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" event={"ID":"6154d545-13b4-434f-8c60-9799415ee4ef","Type":"ContainerDied","Data":"e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77"}
Jul 13 20:50:58.829267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.828870    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"
Jul 13 20:50:58.829267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.828888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx" event={"ID":"6154d545-13b4-434f-8c60-9799415ee4ef","Type":"ContainerDied","Data":"cbde976e162a3e2ba15e15eb70be7fb8398f36b9a41f3d52ba940957f533b479"}
Jul 13 20:50:58.829267 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.828913    2575 scope.go:117] "RemoveContainer" containerID="ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac"
Jul 13 20:50:58.838157 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.838139    2575 scope.go:117] "RemoveContainer" containerID="e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77"
Jul 13 20:50:58.846564 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.846545    2575 scope.go:117] "RemoveContainer" containerID="ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac"
Jul 13 20:50:58.846809 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:58.846792    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac\": container with ID starting with ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac not found: ID does not exist" containerID="ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac"
Jul 13 20:50:58.846875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.846818    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac"} err="failed to get container status \"ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac\": rpc error: code = NotFound desc = could not find container \"ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac\": container with ID starting with ca1f7850975ae977b248c112534da05636fe8245d710748358a240fff83c39ac not found: ID does not exist"
Jul 13 20:50:58.846875 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.846836    2575 scope.go:117] "RemoveContainer" containerID="e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77"
Jul 13 20:50:58.847091 ip-10-0-128-233 kubenswrapper[2575]: E0713 20:50:58.847072    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77\": container with ID starting with e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77 not found: ID does not exist" containerID="e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77"
Jul 13 20:50:58.847189 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.847097    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77"} err="failed to get container status \"e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77\": rpc error: code = NotFound desc = could not find container \"e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77\": container with ID starting with e97bb38c23a9baebf1c08d9f9e92419b4c336c4772b2090b17a1b27a1c2d4b77 not found: ID does not exist"
Jul 13 20:50:58.852144 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.852088    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"]
Jul 13 20:50:58.853393 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.853376    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35019-predictor-86b8b845bd-2l9hx"]
Jul 13 20:50:58.923041 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.922970    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:50:58.966961 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.966932    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f722fc21-f418-4e36-8e1a-97dbc86f7d27-proxy-tls\") pod \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") "
Jul 13 20:50:58.967131 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.966980    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb5pr\" (UniqueName: \"kubernetes.io/projected/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-api-access-wb5pr\") pod \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") "
Jul 13 20:50:58.967131 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.967057    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-rbac-proxy-sar-config\") pod \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\" (UID: \"f722fc21-f418-4e36-8e1a-97dbc86f7d27\") "
Jul 13 20:50:58.967482 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.967456    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f722fc21-f418-4e36-8e1a-97dbc86f7d27" (UID: "f722fc21-f418-4e36-8e1a-97dbc86f7d27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:50:58.969269 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.969247    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-api-access-wb5pr" (OuterVolumeSpecName: "kube-api-access-wb5pr") pod "f722fc21-f418-4e36-8e1a-97dbc86f7d27" (UID: "f722fc21-f418-4e36-8e1a-97dbc86f7d27"). InnerVolumeSpecName "kube-api-access-wb5pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:50:58.969339 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:58.969246    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f722fc21-f418-4e36-8e1a-97dbc86f7d27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f722fc21-f418-4e36-8e1a-97dbc86f7d27" (UID: "f722fc21-f418-4e36-8e1a-97dbc86f7d27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:50:59.067714 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.067678    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5pr\" (UniqueName: \"kubernetes.io/projected/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-api-access-wb5pr\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:59.067714 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.067707    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f722fc21-f418-4e36-8e1a-97dbc86f7d27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:59.067714 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.067717    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f722fc21-f418-4e36-8e1a-97dbc86f7d27-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 20:50:59.130383 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.130348    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" path="/var/lib/kubelet/pods/6154d545-13b4-434f-8c60-9799415ee4ef/volumes"
Jul 13 20:50:59.836078 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.836040    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp" event={"ID":"f722fc21-f418-4e36-8e1a-97dbc86f7d27","Type":"ContainerDied","Data":"a4b162a3a38f3dc639298fd0eb0570ef55f36c77ba3eca9d3d4daa34de48cfd3"}
Jul 13 20:50:59.836078 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.836064    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"
Jul 13 20:50:59.836606 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.836147    2575 scope.go:117] "RemoveContainer" containerID="6c7550bce4ef6267c1ed50477183e2ce61abdb5c652da533085e5aa61afc75c7"
Jul 13 20:50:59.847010 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.846961    2575 scope.go:117] "RemoveContainer" containerID="f1cb0689c5ca871592d6bc4a6c61a3b8e0d894b161a7af55794b2dc64ee4c17e"
Jul 13 20:50:59.853606 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.853587    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"]
Jul 13 20:50:59.857061 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:50:59.857039    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35019-predictor-6647d995d5-4jxvp"]
Jul 13 20:51:01.130188 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:01.130152    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" path="/var/lib/kubelet/pods/f722fc21-f418-4e36-8e1a-97dbc86f7d27/volumes"
Jul 13 20:51:01.763723 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:01.763683    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 20:51:01.820867 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:01.820840    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:51:01.821303 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:01.821281    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:51:01.821433 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:01.821355    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 20:51:01.821631 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:01.821603    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 20:51:02.769446 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:02.769413    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 20:51:11.763195 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:11.763158    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 20:51:11.822339 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:11.822296    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 20:51:11.822535 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:11.822296    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 20:51:12.769252 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:12.769210    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 20:51:21.763535 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:21.763491    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 20:51:21.822082 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:21.822035    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 20:51:21.822230 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:21.822035    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 20:51:22.769915 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:22.769875    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 13 20:51:31.763300 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:31.763261    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 13 20:51:31.821400 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:31.821363    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 20:51:31.821716 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:31.821691    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 20:51:32.770610 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:32.770580    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 20:51:41.764153 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:41.764111    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 20:51:41.822185 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:41.822155    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 20:51:41.822398 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:51:41.822378    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 20:54:15.131535 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:54:15.131503    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:54:15.136538 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:54:15.136516    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:59:15.152732 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:59:15.152633    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 20:59:15.159196 ip-10-0-128-233 kubenswrapper[2575]: I0713 20:59:15.159174    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 21:00:09.748041 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:09.747919    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"]
Jul 13 21:00:09.748580 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:09.748191    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" containerID="cri-o://100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f" gracePeriod=30
Jul 13 21:00:09.748580 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:09.748265    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kube-rbac-proxy" containerID="cri-o://1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d" gracePeriod=30
Jul 13 21:00:09.781505 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:09.781476    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"]
Jul 13 21:00:09.781746 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:09.781723    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" containerID="cri-o://9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787" gracePeriod=30
Jul 13 21:00:09.781824 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:09.781734    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kube-rbac-proxy" containerID="cri-o://7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1" gracePeriod=30
Jul 13 21:00:10.066727 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:10.066636    2575 generic.go:356] "Generic (PLEG): container finished" podID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerID="7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1" exitCode=2
Jul 13 21:00:10.066727 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:10.066714    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" event={"ID":"63603a40-71b1-470e-bc4c-424e7c0f418c","Type":"ContainerDied","Data":"7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1"}
Jul 13 21:00:10.068398 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:10.068366    2575 generic.go:356] "Generic (PLEG): container finished" podID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerID="1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d" exitCode=2
Jul 13 21:00:10.068525 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:10.068431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" event={"ID":"f5fc5723-4d05-4cea-ba54-22cea2393bec","Type":"ContainerDied","Data":"1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d"}
Jul 13 21:00:11.817424 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:11.817383    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 13 21:00:11.817815 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:11.817391    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 13 21:00:11.821947 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:11.821920    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 13 21:00:11.822036 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:11.821919    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 13 21:00:12.999650 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:12.999624    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 21:00:13.000856 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.000836    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9glgj\" (UniqueName: \"kubernetes.io/projected/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-api-access-9glgj\") pod \"f5fc5723-4d05-4cea-ba54-22cea2393bec\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") "
Jul 13 21:00:13.000952 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.000867    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5fc5723-4d05-4cea-ba54-22cea2393bec-proxy-tls\") pod \"f5fc5723-4d05-4cea-ba54-22cea2393bec\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") "
Jul 13 21:00:13.000952 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.000925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-rbac-proxy-sar-config\") pod \"f5fc5723-4d05-4cea-ba54-22cea2393bec\" (UID: \"f5fc5723-4d05-4cea-ba54-22cea2393bec\") "
Jul 13 21:00:13.001279 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.001257    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5fc5723-4d05-4cea-ba54-22cea2393bec" (UID: "f5fc5723-4d05-4cea-ba54-22cea2393bec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:00:13.002892 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.002869    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-api-access-9glgj" (OuterVolumeSpecName: "kube-api-access-9glgj") pod "f5fc5723-4d05-4cea-ba54-22cea2393bec" (UID: "f5fc5723-4d05-4cea-ba54-22cea2393bec"). InnerVolumeSpecName "kube-api-access-9glgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:00:13.003202 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.003179    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fc5723-4d05-4cea-ba54-22cea2393bec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5fc5723-4d05-4cea-ba54-22cea2393bec" (UID: "f5fc5723-4d05-4cea-ba54-22cea2393bec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:00:13.081691 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.081657    2575 generic.go:356] "Generic (PLEG): container finished" podID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerID="100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f" exitCode=0
Jul 13 21:00:13.081869 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.081720    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"
Jul 13 21:00:13.081869 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.081738    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" event={"ID":"f5fc5723-4d05-4cea-ba54-22cea2393bec","Type":"ContainerDied","Data":"100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f"}
Jul 13 21:00:13.081869 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.081839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw" event={"ID":"f5fc5723-4d05-4cea-ba54-22cea2393bec","Type":"ContainerDied","Data":"bedcc9d08ebc39d01d5497c503a57073d1ffdaa8d1a0bb288844ee0516d971fa"}
Jul 13 21:00:13.082085 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.081878    2575 scope.go:117] "RemoveContainer" containerID="1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d"
Jul 13 21:00:13.094201 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.094182    2575 scope.go:117] "RemoveContainer" containerID="100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f"
Jul 13 21:00:13.101559 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.101453    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:00:13.101559 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.101480    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9glgj\" (UniqueName: \"kubernetes.io/projected/f5fc5723-4d05-4cea-ba54-22cea2393bec-kube-api-access-9glgj\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:00:13.101559 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.101531    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5fc5723-4d05-4cea-ba54-22cea2393bec-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:00:13.101974 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.101600    2575 scope.go:117] "RemoveContainer" containerID="1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d"
Jul 13 21:00:13.102252 ip-10-0-128-233 kubenswrapper[2575]: E0713 21:00:13.102163    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d\": container with ID starting with 1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d not found: ID does not exist" containerID="1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d"
Jul 13 21:00:13.102252 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.102195    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d"} err="failed to get container status \"1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d\": rpc error: code = NotFound desc = could not find container \"1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d\": container with ID starting with 1748549d94abe49eea4268b8cb6445f2ce2c214a84d70705afc2aaf870702e7d not found: ID does not exist"
Jul 13 21:00:13.102252 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.102221    2575 scope.go:117] "RemoveContainer" containerID="100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f"
Jul 13 21:00:13.102509 ip-10-0-128-233 kubenswrapper[2575]: E0713 21:00:13.102491    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f\": container with ID starting with 100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f not found: ID does not exist" containerID="100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f"
Jul 13 21:00:13.102550 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.102517    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f"} err="failed to get container status \"100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f\": rpc error: code = NotFound desc = could not find container \"100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f\": container with ID starting with 100d8f947edc4fa7b9f9178ebd934dcc13349c758ae622c985124cb44d3db21f not found: ID does not exist"
Jul 13 21:00:13.103163 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.103145    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"]
Jul 13 21:00:13.107039 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.107018    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-492f6-predictor-6698d5fcc5-m8sgw"]
Jul 13 21:00:13.133406 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.133372    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" path="/var/lib/kubelet/pods/f5fc5723-4d05-4cea-ba54-22cea2393bec/volumes"
Jul 13 21:00:13.220843 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.220819    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 21:00:13.302822 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.302738    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63603a40-71b1-470e-bc4c-424e7c0f418c-proxy-tls\") pod \"63603a40-71b1-470e-bc4c-424e7c0f418c\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") "
Jul 13 21:00:13.302822 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.302796    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ppxx\" (UniqueName: \"kubernetes.io/projected/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-api-access-9ppxx\") pod \"63603a40-71b1-470e-bc4c-424e7c0f418c\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") "
Jul 13 21:00:13.303085 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.302856    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-rbac-proxy-sar-config\") pod \"63603a40-71b1-470e-bc4c-424e7c0f418c\" (UID: \"63603a40-71b1-470e-bc4c-424e7c0f418c\") "
Jul 13 21:00:13.303298 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.303271    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63603a40-71b1-470e-bc4c-424e7c0f418c" (UID: "63603a40-71b1-470e-bc4c-424e7c0f418c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:00:13.305055 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.305035    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-api-access-9ppxx" (OuterVolumeSpecName: "kube-api-access-9ppxx") pod "63603a40-71b1-470e-bc4c-424e7c0f418c" (UID: "63603a40-71b1-470e-bc4c-424e7c0f418c"). InnerVolumeSpecName "kube-api-access-9ppxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:00:13.305117 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.305044    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63603a40-71b1-470e-bc4c-424e7c0f418c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63603a40-71b1-470e-bc4c-424e7c0f418c" (UID: "63603a40-71b1-470e-bc4c-424e7c0f418c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:00:13.403942 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.403914    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:00:13.403942 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.403941    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63603a40-71b1-470e-bc4c-424e7c0f418c-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:00:13.404200 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:13.403955    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppxx\" (UniqueName: \"kubernetes.io/projected/63603a40-71b1-470e-bc4c-424e7c0f418c-kube-api-access-9ppxx\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:00:14.087895 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.087798    2575 generic.go:356] "Generic (PLEG): container finished" podID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerID="9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787" exitCode=0
Jul 13 21:00:14.087895 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.087865    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"
Jul 13 21:00:14.088408 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.087884    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" event={"ID":"63603a40-71b1-470e-bc4c-424e7c0f418c","Type":"ContainerDied","Data":"9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787"}
Jul 13 21:00:14.088408 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.088013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8" event={"ID":"63603a40-71b1-470e-bc4c-424e7c0f418c","Type":"ContainerDied","Data":"864a745253aae85c2c801d0a99df3ae614cdc0f5f7ba8334a9653e9090108a5b"}
Jul 13 21:00:14.088408 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.088045    2575 scope.go:117] "RemoveContainer" containerID="7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1"
Jul 13 21:00:14.096770 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.096749    2575 scope.go:117] "RemoveContainer" containerID="9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787"
Jul 13 21:00:14.103892 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.103864    2575 scope.go:117] "RemoveContainer" containerID="7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1"
Jul 13 21:00:14.104187 ip-10-0-128-233 kubenswrapper[2575]: E0713 21:00:14.104166    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1\": container with ID starting with 7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1 not found: ID does not exist" containerID="7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1"
Jul 13 21:00:14.104256 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.104195    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1"} err="failed to get container status \"7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1\": rpc error: code = NotFound desc = could not find container \"7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1\": container with ID starting with 7f3afdb989e4aa818df8f369e831161e956f19063e3f670ec2e1901a2bc843b1 not found: ID does not exist"
Jul 13 21:00:14.104256 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.104213    2575 scope.go:117] "RemoveContainer" containerID="9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787"
Jul 13 21:00:14.104487 ip-10-0-128-233 kubenswrapper[2575]: E0713 21:00:14.104468    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787\": container with ID starting with 9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787 not found: ID does not exist" containerID="9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787"
Jul 13 21:00:14.104535 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.104493    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787"} err="failed to get container status \"9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787\": rpc error: code = NotFound desc = could not find container \"9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787\": container with ID starting with 9973d49f24f1ddffcaa9564f06ce76df0e389b70a05a15be032b54fd1797f787 not found: ID does not exist"
Jul 13 21:00:14.109653 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.109632    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"]
Jul 13 21:00:14.112819 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:14.112798    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-492f6-predictor-598d856bcd-kxgf8"]
Jul 13 21:00:15.129458 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:00:15.129422    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" path="/var/lib/kubelet/pods/63603a40-71b1-470e-bc4c-424e7c0f418c/volumes"
Jul 13 21:04:15.174978 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:04:15.174874    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 21:04:15.185425 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:04:15.185406    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 21:08:13.770792 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:13.770749    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"]
Jul 13 21:08:13.771422 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:13.771043    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container" containerID="cri-o://42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e" gracePeriod=30
Jul 13 21:08:13.771422 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:13.771097    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kube-rbac-proxy" containerID="cri-o://8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4" gracePeriod=30
Jul 13 21:08:13.820052 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:13.820019    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"]
Jul 13 21:08:13.820349 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:13.820302    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container" containerID="cri-o://deab5066495b770f2d857269d6a7790008ffd0499af249fdf7b66254e9f4492a" gracePeriod=30
Jul 13 21:08:13.820415 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:13.820364    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kube-rbac-proxy" containerID="cri-o://3152710d5b6bbd4e4a982b5461199447fab0dd4dedc0f12dbba76a74594cfed1" gracePeriod=30
Jul 13 21:08:14.839738 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:14.839707    2575 generic.go:356] "Generic (PLEG): container finished" podID="bc85ef38-2865-43c7-a334-96fc8736d535" containerID="3152710d5b6bbd4e4a982b5461199447fab0dd4dedc0f12dbba76a74594cfed1" exitCode=2
Jul 13 21:08:14.840180 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:14.839787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" event={"ID":"bc85ef38-2865-43c7-a334-96fc8736d535","Type":"ContainerDied","Data":"3152710d5b6bbd4e4a982b5461199447fab0dd4dedc0f12dbba76a74594cfed1"}
Jul 13 21:08:14.841570 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:14.841551    2575 generic.go:356] "Generic (PLEG): container finished" podID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerID="8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4" exitCode=2
Jul 13 21:08:14.841668 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:14.841587    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" event={"ID":"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d","Type":"ContainerDied","Data":"8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4"}
Jul 13 21:08:16.826543 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.826515    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 21:08:16.851380 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.851301    2575 generic.go:356] "Generic (PLEG): container finished" podID="bc85ef38-2865-43c7-a334-96fc8736d535" containerID="deab5066495b770f2d857269d6a7790008ffd0499af249fdf7b66254e9f4492a" exitCode=0
Jul 13 21:08:16.851540 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.851376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" event={"ID":"bc85ef38-2865-43c7-a334-96fc8736d535","Type":"ContainerDied","Data":"deab5066495b770f2d857269d6a7790008ffd0499af249fdf7b66254e9f4492a"}
Jul 13 21:08:16.853066 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.853038    2575 generic.go:356] "Generic (PLEG): container finished" podID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerID="42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e" exitCode=0
Jul 13 21:08:16.853193 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.853090    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"
Jul 13 21:08:16.853193 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.853106    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" event={"ID":"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d","Type":"ContainerDied","Data":"42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e"}
Jul 13 21:08:16.853193 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.853162    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" event={"ID":"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d","Type":"ContainerDied","Data":"7a68ff3664cb84328580eef661ad4b18dcdb715c16ee1c7d94eb3048db5c6d0a"}
Jul 13 21:08:16.853296 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.853229    2575 scope.go:117] "RemoveContainer" containerID="8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4"
Jul 13 21:08:16.876051 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.876030    2575 scope.go:117] "RemoveContainer" containerID="42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e"
Jul 13 21:08:16.880823 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.880803    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 21:08:16.883173 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.883153    2575 scope.go:117] "RemoveContainer" containerID="8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4"
Jul 13 21:08:16.883426 ip-10-0-128-233 kubenswrapper[2575]: E0713 21:08:16.883407    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4\": container with ID starting with 8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4 not found: ID does not exist" containerID="8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4"
Jul 13 21:08:16.883493 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.883434    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4"} err="failed to get container status \"8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4\": rpc error: code = NotFound desc = could not find container \"8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4\": container with ID starting with 8e5478358320203b3b4114badccdaa9cfaa45f876cc35d97c2d56c6c88c3d2d4 not found: ID does not exist"
Jul 13 21:08:16.883493 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.883456    2575 scope.go:117] "RemoveContainer" containerID="42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e"
Jul 13 21:08:16.883726 ip-10-0-128-233 kubenswrapper[2575]: E0713 21:08:16.883710    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e\": container with ID starting with 42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e not found: ID does not exist" containerID="42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e"
Jul 13 21:08:16.883774 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.883730    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e"} err="failed to get container status \"42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e\": rpc error: code = NotFound desc = could not find container \"42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e\": container with ID starting with 42f24fdd03d8bf68e1dc95a0534a4ca9eb7db3e6d59fb497cc19e2b8b476706e not found: ID does not exist"
Jul 13 21:08:16.928818 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.928785    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-rbac-proxy-sar-config\") pod \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") "
Jul 13 21:08:16.928963 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.928833    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2snm\" (UniqueName: \"kubernetes.io/projected/bc85ef38-2865-43c7-a334-96fc8736d535-kube-api-access-b2snm\") pod \"bc85ef38-2865-43c7-a334-96fc8736d535\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") "
Jul 13 21:08:16.928963 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.928921    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls\") pod \"bc85ef38-2865-43c7-a334-96fc8736d535\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") "
Jul 13 21:08:16.928963 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.928948    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls\") pod \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") "
Jul 13 21:08:16.929111 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.928977    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65s6m\" (UniqueName: \"kubernetes.io/projected/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-api-access-65s6m\") pod \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\" (UID: \"1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d\") "
Jul 13 21:08:16.929111 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.929023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc85ef38-2865-43c7-a334-96fc8736d535-kube-rbac-proxy-sar-config\") pod \"bc85ef38-2865-43c7-a334-96fc8736d535\" (UID: \"bc85ef38-2865-43c7-a334-96fc8736d535\") "
Jul 13 21:08:16.929240 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.929211    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" (UID: "1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:08:16.929475 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.929451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc85ef38-2865-43c7-a334-96fc8736d535-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc85ef38-2865-43c7-a334-96fc8736d535" (UID: "bc85ef38-2865-43c7-a334-96fc8736d535"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:08:16.931093 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.931066    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc85ef38-2865-43c7-a334-96fc8736d535" (UID: "bc85ef38-2865-43c7-a334-96fc8736d535"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:08:16.931189 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.931131    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-api-access-65s6m" (OuterVolumeSpecName: "kube-api-access-65s6m") pod "1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" (UID: "1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d"). InnerVolumeSpecName "kube-api-access-65s6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:08:16.931587 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.931563    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc85ef38-2865-43c7-a334-96fc8736d535-kube-api-access-b2snm" (OuterVolumeSpecName: "kube-api-access-b2snm") pod "bc85ef38-2865-43c7-a334-96fc8736d535" (UID: "bc85ef38-2865-43c7-a334-96fc8736d535"). InnerVolumeSpecName "kube-api-access-b2snm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:08:16.931587 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:16.931569    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" (UID: "1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:08:17.030337 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.030305    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:08:17.030337 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.030332    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2snm\" (UniqueName: \"kubernetes.io/projected/bc85ef38-2865-43c7-a334-96fc8736d535-kube-api-access-b2snm\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:08:17.030337 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.030343    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc85ef38-2865-43c7-a334-96fc8736d535-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:08:17.030561 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.030352    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-proxy-tls\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:08:17.030561 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.030362    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65s6m\" (UniqueName: \"kubernetes.io/projected/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d-kube-api-access-65s6m\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:08:17.030561 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.030372    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc85ef38-2865-43c7-a334-96fc8736d535-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-233.ec2.internal\" DevicePath \"\""
Jul 13 21:08:17.169322 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.169294    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"]
Jul 13 21:08:17.173245 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.173220    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw"]
Jul 13 21:08:17.758517 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.758470    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-575ad-predictor-9f95b5ddf-p6vcw" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 13 21:08:17.859559 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.859476    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6" event={"ID":"bc85ef38-2865-43c7-a334-96fc8736d535","Type":"ContainerDied","Data":"0e4b510808615739599cae39c8177fb4817e366fc9813302a7a176b89d936e4a"}
Jul 13 21:08:17.859559 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.859522    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"
Jul 13 21:08:17.859559 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.859542    2575 scope.go:117] "RemoveContainer" containerID="3152710d5b6bbd4e4a982b5461199447fab0dd4dedc0f12dbba76a74594cfed1"
Jul 13 21:08:17.867663 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.867645    2575 scope.go:117] "RemoveContainer" containerID="deab5066495b770f2d857269d6a7790008ffd0499af249fdf7b66254e9f4492a"
Jul 13 21:08:17.877082 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.877057    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"]
Jul 13 21:08:17.880563 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:17.880540    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-575ad-predictor-7d77b767d4-xbql6"]
Jul 13 21:08:19.129652 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:19.129618    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" path="/var/lib/kubelet/pods/1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d/volumes"
Jul 13 21:08:19.130064 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:19.130049    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" path="/var/lib/kubelet/pods/bc85ef38-2865-43c7-a334-96fc8736d535/volumes"
Jul 13 21:08:46.468387 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468303    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xlsxm/must-gather-x8z64"]
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468582    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468595    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468606    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468612    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468622    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468628    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468635    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468641    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468649    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468655    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468661    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468666    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468674    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468678    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468685    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468689    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468695    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468700    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468707    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468712    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468717    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468722    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468729    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468733    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468772    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468780    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468785    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468791    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468797    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468802    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6154d545-13b4-434f-8c60-9799415ee4ef" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468808    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fc5723-4d05-4cea-ba54-22cea2393bec" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468814    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="63603a40-71b1-470e-bc4c-424e7c0f418c" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468819    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f722fc21-f418-4e36-8e1a-97dbc86f7d27" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468824    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc85ef38-2865-43c7-a334-96fc8736d535" containerName="kube-rbac-proxy"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468830    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kserve-container"
Jul 13 21:08:46.468878 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.468835    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf0ceca-cfb7-49b1-9f7c-4e9bff37a15d" containerName="kube-rbac-proxy"
Jul 13 21:08:46.471535 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.471516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.474188 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.474166    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlsxm\"/\"openshift-service-ca.crt\""
Jul 13 21:08:46.474317 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.474168    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlsxm\"/\"kube-root-ca.crt\""
Jul 13 21:08:46.474317 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.474172    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xlsxm\"/\"default-dockercfg-nr2xr\""
Jul 13 21:08:46.479466 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.479444    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlsxm/must-gather-x8z64"]
Jul 13 21:08:46.559398 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.559354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnh5\" (UniqueName: \"kubernetes.io/projected/4e40da1c-eb3b-4d36-8bce-73eede5eeb53-kube-api-access-5dnh5\") pod \"must-gather-x8z64\" (UID: \"4e40da1c-eb3b-4d36-8bce-73eede5eeb53\") " pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.559398 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.559402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4e40da1c-eb3b-4d36-8bce-73eede5eeb53-must-gather-output\") pod \"must-gather-x8z64\" (UID: \"4e40da1c-eb3b-4d36-8bce-73eede5eeb53\") " pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.660077 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.660040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnh5\" (UniqueName: \"kubernetes.io/projected/4e40da1c-eb3b-4d36-8bce-73eede5eeb53-kube-api-access-5dnh5\") pod \"must-gather-x8z64\" (UID: \"4e40da1c-eb3b-4d36-8bce-73eede5eeb53\") " pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.660260 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.660085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4e40da1c-eb3b-4d36-8bce-73eede5eeb53-must-gather-output\") pod \"must-gather-x8z64\" (UID: \"4e40da1c-eb3b-4d36-8bce-73eede5eeb53\") " pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.660397 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.660380    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4e40da1c-eb3b-4d36-8bce-73eede5eeb53-must-gather-output\") pod \"must-gather-x8z64\" (UID: \"4e40da1c-eb3b-4d36-8bce-73eede5eeb53\") " pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.668156 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.668133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnh5\" (UniqueName: \"kubernetes.io/projected/4e40da1c-eb3b-4d36-8bce-73eede5eeb53-kube-api-access-5dnh5\") pod \"must-gather-x8z64\" (UID: \"4e40da1c-eb3b-4d36-8bce-73eede5eeb53\") " pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.781175 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.781091    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlsxm/must-gather-x8z64"
Jul 13 21:08:46.909136 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.909110    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlsxm/must-gather-x8z64"]
Jul 13 21:08:46.911518 ip-10-0-128-233 kubenswrapper[2575]: W0713 21:08:46.911487    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e40da1c_eb3b_4d36_8bce_73eede5eeb53.slice/crio-45c88ea7f644e7cda2877a087a9025529164bbff09cc53191d8d995a15de95a1 WatchSource:0}: Error finding container 45c88ea7f644e7cda2877a087a9025529164bbff09cc53191d8d995a15de95a1: Status 404 returned error can't find the container with id 45c88ea7f644e7cda2877a087a9025529164bbff09cc53191d8d995a15de95a1
Jul 13 21:08:46.913303 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.913288    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 21:08:46.968464 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:46.968428    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlsxm/must-gather-x8z64" event={"ID":"4e40da1c-eb3b-4d36-8bce-73eede5eeb53","Type":"ContainerStarted","Data":"45c88ea7f644e7cda2877a087a9025529164bbff09cc53191d8d995a15de95a1"}
Jul 13 21:08:47.976975 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:47.976942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlsxm/must-gather-x8z64" event={"ID":"4e40da1c-eb3b-4d36-8bce-73eede5eeb53","Type":"ContainerStarted","Data":"3ede580e0ef6f5b6465ae29b760bd830f8699f5ddc6d00c18d6bd726739cbac5"}
Jul 13 21:08:47.977374 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:47.977009    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlsxm/must-gather-x8z64" event={"ID":"4e40da1c-eb3b-4d36-8bce-73eede5eeb53","Type":"ContainerStarted","Data":"241c56a531924a333c3e784b3fd83f9802eaba9938e5b88d36a812c064f64fad"}
Jul 13 21:08:47.993037 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:47.992957    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xlsxm/must-gather-x8z64" podStartSLOduration=1.212497721 podStartE2EDuration="1.992937705s" totalImagesPullingTime="780.439984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:08:47.99076787 +0000 UTC m=+2973.343376713" watchObservedRunningTime="2026-07-13 21:08:47.992937705 +0000 UTC m=+2973.345546548"
Jul 13 21:08:49.029502 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:49.029470    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hqxqb_636333a8-2370-4e1f-acc9-4ad49e62fcfc/global-pull-secret-syncer/0.log"
Jul 13 21:08:49.203691 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:49.203663    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jnrr6_3f4bbc94-91c8-4a0d-bc00-ea9be56778d2/konnectivity-agent/0.log"
Jul 13 21:08:49.255921 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:49.255893    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-233.ec2.internal_e3a758c83d40d2b976cf492c86764c97/haproxy/0.log"
Jul 13 21:08:53.569269 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:53.569237    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fnm95_15f8d971-bcd4-4779-b5af-c13f76c13dd9/node-exporter/0.log"
Jul 13 21:08:53.596079 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:53.596045    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fnm95_15f8d971-bcd4-4779-b5af-c13f76c13dd9/kube-rbac-proxy/0.log"
Jul 13 21:08:53.622338 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:53.622291    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fnm95_15f8d971-bcd4-4779-b5af-c13f76c13dd9/init-textfile/0.log"
Jul 13 21:08:55.631032 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:55.631005    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-stzmg_dab114e9-37f1-4dae-a99d-9a8533cddcca/networking-console-plugin/0.log"
Jul 13 21:08:57.401364 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.401325    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"]
Jul 13 21:08:57.407503 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.407475    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.413253 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.413227    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"]
Jul 13 21:08:57.556266 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.556231    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-proc\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.556624 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.556597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8wp\" (UniqueName: \"kubernetes.io/projected/5ef59bf8-73fa-40ca-baba-8ae054935f6b-kube-api-access-kj8wp\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.556873 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.556828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-sys\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.557061 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.557042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-lib-modules\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.557246 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.557228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-podres\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658127 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-sys\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658127 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-lib-modules\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658307 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-sys\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658307 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-podres\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658307 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-lib-modules\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658307 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-proc\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658307 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8wp\" (UniqueName: \"kubernetes.io/projected/5ef59bf8-73fa-40ca-baba-8ae054935f6b-kube-api-access-kj8wp\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658468 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-proc\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.658468 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.658333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ef59bf8-73fa-40ca-baba-8ae054935f6b-podres\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.666864 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.666825    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8wp\" (UniqueName: \"kubernetes.io/projected/5ef59bf8-73fa-40ca-baba-8ae054935f6b-kube-api-access-kj8wp\") pod \"perf-node-gather-daemonset-hkww2\" (UID: \"5ef59bf8-73fa-40ca-baba-8ae054935f6b\") " pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.724010 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.723951    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:57.886174 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:57.886080    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"]
Jul 13 21:08:58.025917 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.025885    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2" event={"ID":"5ef59bf8-73fa-40ca-baba-8ae054935f6b","Type":"ContainerStarted","Data":"b287a1193ee2ce97f0834352c1c6aec8432c1f696bc8438517f147898843fcf5"}
Jul 13 21:08:58.026101 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.025940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2" event={"ID":"5ef59bf8-73fa-40ca-baba-8ae054935f6b","Type":"ContainerStarted","Data":"f0b7e6b731ff7d0781d62ba80455d9dd6e1b09d101cc3728795ef2f1f2691e30"}
Jul 13 21:08:58.026101 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.026015    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:08:58.041640 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.041583    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2" podStartSLOduration=1.041561341 podStartE2EDuration="1.041561341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:08:58.041311416 +0000 UTC m=+2983.393920261" watchObservedRunningTime="2026-07-13 21:08:58.041561341 +0000 UTC m=+2983.394170187"
Jul 13 21:08:58.116164 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.116137    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lrsdq_6dd0dafe-9870-40d4-b882-724ab09f3395/dns/0.log"
Jul 13 21:08:58.139586 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.139533    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lrsdq_6dd0dafe-9870-40d4-b882-724ab09f3395/kube-rbac-proxy/0.log"
Jul 13 21:08:58.197488 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.197463    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzfvp_5df6f3a4-09b8-49bf-b87c-b8bc60e31c6e/dns-node-resolver/0.log"
Jul 13 21:08:58.809404 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:08:58.809376    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-68pm9_6498e489-3b9d-4ca6-8191-b2cd12b9c37a/node-ca/0.log"
Jul 13 21:09:00.004793 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:00.004759    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hdctk_08728e65-2a79-442c-b3f8-7704552630e2/serve-healthcheck-canary/0.log"
Jul 13 21:09:00.473093 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:00.473066    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbtzv_4e3d19be-0c48-4ee6-8ec5-26059fc9e318/kube-rbac-proxy/0.log"
Jul 13 21:09:00.497622 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:00.497594    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbtzv_4e3d19be-0c48-4ee6-8ec5-26059fc9e318/exporter/0.log"
Jul 13 21:09:00.521278 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:00.521253    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbtzv_4e3d19be-0c48-4ee6-8ec5-26059fc9e318/extractor/0.log"
Jul 13 21:09:02.703627 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:02.703597    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-68b649c569-ztbrc_89811c72-6825-4c59-9b3e-7be3d1319e7e/manager/0.log"
Jul 13 21:09:03.015915 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:03.015819    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-9w8qp_7af263c0-d8a2-4efd-8c92-d885b7f43cb2/manager/0.log"
Jul 13 21:09:03.139527 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:03.139501    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-z64s4_cac9a4d4-3a0a-4474-b1a4-60b4ec9d3155/seaweedfs/0.log"
Jul 13 21:09:04.040618 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:04.040589    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xlsxm/perf-node-gather-daemonset-hkww2"
Jul 13 21:09:07.037037 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:07.036904    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-tfhfg_83ba3960-58a8-4464-bbcf-6dd4e9d95911/migrator/0.log"
Jul 13 21:09:07.059227 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:07.059197    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-tfhfg_83ba3960-58a8-4464-bbcf-6dd4e9d95911/graceful-termination/0.log"
Jul 13 21:09:08.302537 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.302511    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hc99x_28f13592-025b-493b-a761-aab0e1bfda3c/kube-multus-additional-cni-plugins/0.log"
Jul 13 21:09:08.327282 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.327252    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hc99x_28f13592-025b-493b-a761-aab0e1bfda3c/egress-router-binary-copy/0.log"
Jul 13 21:09:08.347602 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.347573    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hc99x_28f13592-025b-493b-a761-aab0e1bfda3c/cni-plugins/0.log"
Jul 13 21:09:08.367353 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.367327    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hc99x_28f13592-025b-493b-a761-aab0e1bfda3c/bond-cni-plugin/0.log"
Jul 13 21:09:08.387257 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.387227    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hc99x_28f13592-025b-493b-a761-aab0e1bfda3c/routeoverride-cni/0.log"
Jul 13 21:09:08.407311 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.407244    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hc99x_28f13592-025b-493b-a761-aab0e1bfda3c/whereabouts-cni-bincopy/0.log"
Jul 13 21:09:08.428287 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.428212    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hc99x_28f13592-025b-493b-a761-aab0e1bfda3c/whereabouts-cni/0.log"
Jul 13 21:09:08.807495 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.807421    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v25wz_baf87b37-94c8-4788-b01b-bd97a1ec5c63/kube-multus/0.log"
Jul 13 21:09:08.903146 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.903116    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nrn7d_1347ce06-e218-4be5-9dd2-5db94f75ff35/network-metrics-daemon/0.log"
Jul 13 21:09:08.920051 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:08.920021    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nrn7d_1347ce06-e218-4be5-9dd2-5db94f75ff35/kube-rbac-proxy/0.log"
Jul 13 21:09:10.042494 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.042461    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-controller/0.log"
Jul 13 21:09:10.059027 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.058978    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/0.log"
Jul 13 21:09:10.059805 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.059787    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovn-acl-logging/1.log"
Jul 13 21:09:10.080853 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.080829    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/kube-rbac-proxy-node/0.log"
Jul 13 21:09:10.109447 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.109426    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/kube-rbac-proxy-ovn-metrics/0.log"
Jul 13 21:09:10.134699 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.134677    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/northd/0.log"
Jul 13 21:09:10.160824 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.160792    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/nbdb/0.log"
Jul 13 21:09:10.185821 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.185797    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/sbdb/0.log"
Jul 13 21:09:10.258783 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:10.258748    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cwl74_1de3016b-df9e-4924-8f6b-5f42bbc831d8/ovnkube-controller/0.log"
Jul 13 21:09:11.270270 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:11.270238    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-r557g_9556dff5-b188-4ff9-aaf2-dc315a1ba735/network-check-target-container/0.log"
Jul 13 21:09:12.181600 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:12.181569    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9tj22_7a052240-fd7b-4102-9bb4-b2b9506f9961/iptables-alerter/0.log"
Jul 13 21:09:12.889496 ip-10-0-128-233 kubenswrapper[2575]: I0713 21:09:12.889462    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-lxj57_6b9111d6-4502-4d2d-8594-89f4281983f3/tuned/0.log"